{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Pre-training BERT from scratch with cloud TPU",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "TPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E2XB_l-Hgzq_",
        "colab_type": "text"
      },
      "source": [
        "# Pre-training BERT from scratch with cloud TPU"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vZPgpRl5g2e2",
        "colab_type": "text"
      },
      "source": [
        "In this experiment, we will be pre-training a state-of-the-art Natural Language Understanding model [BERT](https://arxiv.org/abs/1810.04805.) on arbitrary text data using Google Cloud infrastructure.\n",
        "\n",
        "This guide covers all stages of the procedure, including:\n",
        "\n",
        "1. Setting up the training environment\n",
        "2. Downloading raw text data\n",
        "3. Preprocessing text data\n",
        "4. Learning a new vocabulary\n",
        "5. Creating sharded pre-training data\n",
        "6. Setting up GCS storage for data and model\n",
        "7. Training the model on a cloud TPU\n",
        "\n",
        "For persistent storage of training data and model, you will require a Google Cloud Storage bucket. \n",
        "Please follow the [Google Cloud TPU quickstart](https://cloud.google.com/tpu/docs/quickstart) to create a GCP account and GCS bucket. New Google Cloud users have [$300 free credit](https://cloud.google.com/free/) to get started with any GCP product. \n",
        "\n",
        "Steps 1-5 of this tutorial can be run without a GCS bucket for demonstration purposes. In that case, however, you will not be able to train the model.\n",
        "\n",
        "**Note** \n",
        "The only parameter you *really have to set* is BUCKET_NAME in steps 5 and 6. Everything else has default values which should work for most use-cases.\n",
        "\n",
        "**Note** \n",
        "Pre-training a BERT-Base model on a TPUv2 will take about 54 hours. Google Colab is not designed for executing such long-running jobs and will interrupt the training process every 8 hours or so. For uninterrupted training, consider using a preemptible TPUv2 instance. \n",
        "\n",
        "That said, at the time of writing (09.05.2019), with a Colab TPU, pre-training a BERT model from scratch can be achieved at a negligible cost of storing the said model and data in GCS  (~1 USD).\n",
        "\n",
        "Now, let's get to business."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ODimOhBR05yR",
        "colab_type": "text"
      },
      "source": [
        "MIT License\n",
        "\n",
        "Copyright (c) [2019] [Antyukhov Denis Olegovich]\n",
        "\n",
        "Permission is hereby granted, free of charge, to any person obtaining a copy\n",
        "of this software and associated documentation files (the \"Software\"), to deal\n",
        "in the Software without restriction, including without limitation the rights\n",
        "to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n",
        "copies of the Software, and to permit persons to whom the Software is\n",
        "furnished to do so, subject to the following conditions:\n",
        "\n",
        "The above copyright notice and this permission notice shall be included in all\n",
        "copies or substantial portions of the Software.\n",
        "\n",
        "THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
        "IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
        "FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n",
        "AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
        "LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n",
        "OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n",
        "SOFTWARE."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fjad5jsr9YaM",
        "colab_type": "text"
      },
      "source": [
        "## Step 1: setting up training environment\n",
        "First and foremost, we get the packages required to train the model. \n",
        "The Jupyter environment allows executing bash commands directly from the notebook by using an exclamation mark ‘!’. I will be exploiting this approach to make use of several other bash commands throughout the experiment.\n",
        "\n",
        "Now, let’s import the packages and authorize ourselves in Google Cloud."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9S4CiOh3RzFW",
        "colab_type": "code",
        "outputId": "c705c95e-ba1f-4c73-c545-6006bed6cb71",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 360
        }
      },
      "source": [
        "!pip install sentencepiece\n",
        "!git clone https://github.com/google-research/bert\n",
        "\n",
        "import os\n",
        "import sys\n",
        "import json\n",
        "import nltk\n",
        "import random\n",
        "import logging\n",
        "import tensorflow as tf\n",
        "import sentencepiece as spm\n",
        "\n",
        "from glob import glob\n",
        "from google.colab import auth, drive\n",
        "from tensorflow.keras.utils import Progbar\n",
        "\n",
        "sys.path.append(\"bert\")\n",
        "\n",
        "from bert import modeling, optimization, tokenization\n",
        "from bert.run_pretraining import input_fn_builder, model_fn_builder\n",
        "\n",
        "auth.authenticate_user()\n",
        "  \n",
        "# configure logging\n",
        "log = logging.getLogger('tensorflow')\n",
        "log.setLevel(logging.INFO)\n",
        "\n",
        "# create formatter and add it to the handlers\n",
        "formatter = logging.Formatter('%(asctime)s :  %(message)s')\n",
        "sh = logging.StreamHandler()\n",
        "sh.setLevel(logging.INFO)\n",
        "sh.setFormatter(formatter)\n",
        "log.handlers = [sh]\n",
        "\n",
        "if 'COLAB_TPU_ADDR' in os.environ:\n",
        "  log.info(\"Using TPU runtime\")\n",
        "  USE_TPU = True\n",
        "  TPU_ADDRESS = 'grpc://' + os.environ['COLAB_TPU_ADDR']\n",
        "\n",
        "  with tf.Session(TPU_ADDRESS) as session:\n",
        "    log.info('TPU address is ' + TPU_ADDRESS)\n",
        "    # Upload credentials to TPU.\n",
        "    with open('/content/adc.json', 'r') as f:\n",
        "      auth_info = json.load(f)\n",
        "    tf.contrib.cloud.configure_gcs(session, credentials=auth_info)\n",
        "    \n",
        "else:\n",
        "  log.warning('Not connected to TPU runtime')\n",
        "  USE_TPU = False"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting sentencepiece\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/00/95/7f357995d5eb1131aa2092096dca14a6fc1b1d2860bd99c22a612e1d1019/sentencepiece-0.1.82-cp36-cp36m-manylinux1_x86_64.whl (1.0MB)\n",
            "\u001b[K     |████████████████████████████████| 1.0MB 3.5MB/s \n",
            "\u001b[?25hInstalling collected packages: sentencepiece\n",
            "Successfully installed sentencepiece-0.1.82\n",
            "Cloning into 'bert'...\n",
            "remote: Enumerating objects: 325, done.\u001b[K\n",
            "remote: Total 325 (delta 0), reused 0 (delta 0), pack-reused 325\u001b[K\n",
            "Receiving objects: 100% (325/325), 234.48 KiB | 3.17 MiB/s, done.\n",
            "Resolving deltas: 100% (186/186), done.\n",
            "\n",
            "WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
            "For more information, please see:\n",
            "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
            "  * https://github.com/tensorflow/addons\n",
            "If you depend on functionality not listed there, please file an issue.\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "2019-05-09 13:19:25,846 :  Using TPU runtime\n",
            "2019-05-09 13:19:25,849 :  TPU address is grpc://10.60.104.26:8470\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vvfNt0mVZ33n",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PGVXMoC-aMy1",
        "colab_type": "text"
      },
      "source": [
        "## Step 2: getting the data\n",
        "\n",
        "We begin with obtaining a corpus of raw text data. For this experiment, we will be using the [OpenSubtitles](http://www.opensubtitles.org/) dataset, which is available for 65 languages [here](http://opus.nlpl.eu/OpenSubtitles-v2016.php). \n",
        "\n",
        "Unlike more common text datasets (like Wikipedia) it does not require any complex pre-processing. It also comes pre-formatted with one sentence per line.\n",
        "\n",
        "Feel free to use the dataset for your language instead by changing the language code (en) below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FotFkkshbdvK",
        "colab_type": "code",
        "outputId": "2fc89d88-1af1-4bc6-8130-7d92d043c3d0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 479
        }
      },
      "source": [
        "AVAILABLE =  {'af','ar','bg','bn','br','bs','ca','cs',\n",
        "              'da','de','el','en','eo','es','et','eu',\n",
        "              'fa','fi','fr','gl','he','hi','hr','hu',\n",
        "              'hy','id','is','it','ja','ka','kk','ko',\n",
        "              'lt','lv','mk','ml','ms','nl','no','pl',\n",
        "              'pt','pt_br','ro','ru','si','sk','sl','sq',\n",
        "              'sr','sv','ta','te','th','tl','tr','uk',\n",
        "              'ur','vi','ze_en','ze_zh','zh','zh_cn',\n",
        "              'zh_en','zh_tw','zh_zh'}\n",
        "\n",
        "LANG_CODE = \"en\" #@param {type:\"string\"}\n",
        "\n",
        "assert LANG_CODE in AVAILABLE, \"Invalid language code selected\"\n",
        "\n",
        "!wget http://opus.nlpl.eu/download.php?f=OpenSubtitles/v2016/mono/OpenSubtitles.raw.'$LANG_CODE'.gz -O dataset.txt.gz\n",
        "!gzip -d dataset.txt.gz\n",
        "!tail dataset.txt"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2019-05-09 13:21:14--  http://opus.nlpl.eu/download.php?f=OpenSubtitles/v2016/mono/OpenSubtitles.raw.en.gz\n",
            "Resolving opus.nlpl.eu (opus.nlpl.eu)... 193.166.25.9\n",
            "Connecting to opus.nlpl.eu (opus.nlpl.eu)|193.166.25.9|:80... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://object.pouta.csc.fi/OPUS-OpenSubtitles/v2016/mono/en.txt.gz [following]\n",
            "--2019-05-09 13:21:15--  https://object.pouta.csc.fi/OPUS-OpenSubtitles/v2016/mono/en.txt.gz\n",
            "Resolving object.pouta.csc.fi (object.pouta.csc.fi)... 86.50.254.0, 86.50.254.1\n",
            "Connecting to object.pouta.csc.fi (object.pouta.csc.fi)|86.50.254.0|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 2906709304 (2.7G) [application/gzip]\n",
            "Saving to: ‘dataset.txt.gz’\n",
            "\n",
            "dataset.txt.gz      100%[===================>]   2.71G  20.5MB/s    in 2m 16s  \n",
            "\n",
            "2019-05-09 13:23:31 (20.5 MB/s) - ‘dataset.txt.gz’ saved [2906709304/2906709304]\n",
            "\n",
            "The astronomers run at full speed turning around each time they are pressed too closely and reducing the fragile beings to dust.\n",
            "At last, the astronomers have found the shell and quickly shut themselves in the interior.\n",
            "Thanks to the advance, they have succeeded in getting over their adversaries.\n",
            "Only one, the president, has been left behind.\n",
            "He rushes to the rope which hangs from the point of the shell and letting himself slide down the rope he gives it an impetus which causes the shell to fall off the edge of the moon.\n",
            "The shell falls with sickening rapidity.\n",
            "The sea appears.\n",
            "The shell balances and thanks to the hermetically sealed air in its interior rises slowly to the surface.\n",
            "The shell is picked up by a steamer which tows it to port.\n",
            "The Mayor welcomes the astronomers with a speech, and the general ovation awaits their happy return.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nb5TPsPOppn0",
        "colab_type": "text"
      },
      "source": [
        "For demonstration purposes, we will only use a small fraction of the whole corpus for this experiment. \n",
        "\n",
        "When training the real model, make sure to uncheck the DEMO_MODE checkbox to use a 100x larger dataset.\n",
        "\n",
        "Rest assured, 100M lines are perfectly sufficient to train a reasonably good BERT-base model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UDR5Z1MDgB1g",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "DEMO_MODE = True #@param {type:\"boolean\"}\n",
        "\n",
        "if DEMO_MODE:\n",
        "  CORPUS_SIZE = 1000000\n",
        "else:\n",
        "  CORPUS_SIZE = 100000000 #@param {type: \"integer\"}\n",
        "  \n",
        "!(head -n $CORPUS_SIZE dataset.txt) > subdataset.txt\n",
        "!mv subdataset.txt dataset.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oRQd4-v0nQqH",
        "colab_type": "text"
      },
      "source": [
        "## Step 3: preprocessing text\n",
        "\n",
        "The raw text data we have downloaded contains punсtuation, uppercase letters and non-UTF symbols which we will remove before proceeding. During inference, we will apply the same normalization procedure to new data.\n",
        "\n",
        "If your use-case requires different preprocessing (e.g. if uppercase letters or punctuation are expected during inference), feel free to modify the function below to accomodate for your needs."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iCi2oSdInRkC",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "regex_tokenizer = nltk.RegexpTokenizer(\"\\w+\")\n",
        "\n",
        "def normalize_text(text):\n",
        "  # lowercase text\n",
        "  text = str(text).lower()\n",
        "  # remove non-UTF\n",
        "  text = text.encode(\"utf-8\", \"ignore\").decode()\n",
        "  # remove punktuation symbols\n",
        "  text = \" \".join(regex_tokenizer.tokenize(text))\n",
        "  return text\n",
        "\n",
        "def count_lines(filename):\n",
        "  count = 0\n",
        "  with open(filename) as fi:\n",
        "    for line in fi:\n",
        "      count += 1\n",
        "  return count"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rYtwtDnesaQQ",
        "colab_type": "text"
      },
      "source": [
        "Check how that works."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0gngtEZWqVhY",
        "colab_type": "code",
        "outputId": "922574d3-a36e-4565-97ea-a236c2f2c870",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "normalize_text('Thanks to the advance, they have succeeded in getting over their adversaries.')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'thanks to the advance they have succeeded in getting over their adversaries'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zY-Kvnx6sUFS",
        "colab_type": "text"
      },
      "source": [
        "Apply normalization to the whole dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "myjxQe5awo1v",
        "colab_type": "code",
        "outputId": "e38b5b20-1d88-4fca-d033-007e62bcd075",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "RAW_DATA_FPATH = \"dataset.txt\" #@param {type: \"string\"}\n",
        "PRC_DATA_FPATH = \"proc_dataset.txt\" #@param {type: \"string\"}\n",
        "\n",
        "# apply normalization to the dataset\n",
        "# this will take a minute or two\n",
        "\n",
        "total_lines = count_lines(RAW_DATA_FPATH)\n",
        "bar = Progbar(total_lines)\n",
        "\n",
        "with open(RAW_DATA_FPATH,encoding=\"utf-8\") as fi:\n",
        "  with open(PRC_DATA_FPATH, \"w\",encoding=\"utf-8\") as fo:\n",
        "    for l in fi:\n",
        "      fo.write(normalize_text(l)+\"\\n\")\n",
        "      bar.add(1)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "1000000/1000000 [==============================] - 6s 6us/step\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O3A64RZjwo9k",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zVO9EnUwrluQ",
        "colab_type": "text"
      },
      "source": [
        "## Step 4: building the vocabulary\n",
        "\n",
        "For the next step, we will learn a new vocabulary that we will use to represent our dataset. \n",
        "\n",
        "The BERT paper uses a WordPiece tokenizer, which is not available in opensource. Instead, we will be using SentencePiece tokenizer in unigram mode. While it is not directly compatible with BERT, with a small hack we can make it work.\n",
        "\n",
        "SentencePiece requires quite a lot of RAM, so running it on the full dataset in Colab will crash the kernel. To avoid this, we will randomly subsample a fraction of the dataset for building the vocabulary. Another option would be to use a machine with more RAM for this step - that decision is up to you.\n",
        "\n",
        "Also, SentencePiece adds BOS and EOS control symbols to the vocabulary by default. We disable them explicitly by setting their indices to -1.\n",
        "\n",
        "The typical values for VOC_SIZE are somewhere in between 32000 and 128000. We reserve NUM_PLACEHOLDERS tokens in case one wants to update the vocabulary and fine-tune the model after the pre-training phase is finished."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "18nn6eW_s-fV",
        "colab_type": "code",
        "outputId": "cb2f04b0-ebd5-40cf-e53f-0079d303c625",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "MODEL_PREFIX = \"tokenizer\" #@param {type: \"string\"}\n",
        "VOC_SIZE = 32000 #@param {type:\"integer\"}\n",
        "SUBSAMPLE_SIZE = 12800000 #@param {type:\"integer\"}\n",
        "NUM_PLACEHOLDERS = 256 #@param {type:\"integer\"}\n",
        "\n",
        "SPM_COMMAND = ('--input={} --model_prefix={} '\n",
        "               '--vocab_size={} --input_sentence_size={} '\n",
        "               '--shuffle_input_sentence=true ' \n",
        "               '--bos_id=-1 --eos_id=-1').format(\n",
        "               PRC_DATA_FPATH, MODEL_PREFIX, \n",
        "               VOC_SIZE - NUM_PLACEHOLDERS, SUBSAMPLE_SIZE)\n",
        "\n",
        "spm.SentencePieceTrainer.Train(SPM_COMMAND)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mAowCoH2u1iZ",
        "colab_type": "text"
      },
      "source": [
        "Now let's see how we can make SentencePiece tokenizer work for the BERT model. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4OLaw7kPW3he",
        "colab_type": "text"
      },
      "source": [
        "Below is a sentence tokenized using the WordPiece vocabulary from a pretrained English [BERT-base](https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-12_H-768_A-12.zip) model from the official [repo](https://github.com/google-research/bert). "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kwHAp_Gh5OPK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "testcase = \"Colorless geothermal substations are generating furiously\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NyEGAVl_5YRD",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "\n",
        "```\n",
        ">>> wordpiece.tokenize(\"Colorless geothermal substations are generating furiously\")\n",
        "\n",
        "['color',\n",
        " '##less',\n",
        " 'geo',\n",
        " '##thermal',\n",
        " 'sub',\n",
        " '##station',\n",
        " '##s',\n",
        " 'are',\n",
        " 'generating',\n",
        " 'furiously']\n",
        "```\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BNiLdWXTh9cj",
        "colab_type": "text"
      },
      "source": [
        "As we can see, the WordPiece tokenizer prepends the subwords which occur in the middle of words with '##'. The subwords occurring at the beginning of words are unchanged. If the subword occurs both in the beginning and in the middle of words, both versions (with and without '##') are added to the vocabulary.\n",
        "\n",
        "Now let's have a look at the vocabulary that the SentencePiece tokenizer has learned."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H8_ebLxqTnWu",
        "colab_type": "code",
        "outputId": "bcb062f3-5261-42d1-9617-205b6bc6d6ff",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "!ls"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "adc.json  dataset.txt\t    sample_data      tokenizer.vocab\n",
            "bert\t  proc_dataset.txt  tokenizer.model\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nYlBqiv5UD-j",
        "colab_type": "text"
      },
      "source": [
        "SentencePiece has created two files: tokenizer.model and tokenizer.vocab. Let's have a look at the learned vocabulary:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WDJ9QmNMUEQf",
        "colab_type": "code",
        "outputId": "d1cbad8b-6e98-4e05-faa9-1b3642a80f1a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 187
        }
      },
      "source": [
        "!head -n 10 tokenizer.vocab"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<unk>\t0\n",
            "▁you\t-3.2342\n",
            "▁i\t-3.2821\n",
            "▁the\t-3.56375\n",
            "▁s\t-3.84955\n",
            "▁to\t-3.87601\n",
            "▁a\t-3.9102\n",
            "▁it\t-3.97593\n",
            "▁t\t-4.25729\n",
            "▁and\t-4.32686\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yBsURk_h5jw4",
        "colab_type": "code",
        "outputId": "3a49e4df-b3af-4c46-9f66-0b32f1d265f4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "def read_sentencepiece_vocab(filepath):\n",
        "  voc = []\n",
        "  with open(filepath, encoding='utf-8') as fi:\n",
        "    for line in fi:\n",
        "      voc.append(line.split(\"\\t\")[0])\n",
        "  # skip the first <unk> token\n",
        "  voc = voc[1:]\n",
        "  return voc\n",
        "\n",
        "snt_vocab = read_sentencepiece_vocab(\"{}.vocab\".format(MODEL_PREFIX))\n",
        "print(\"Learnt vocab size: {}\".format(len(snt_vocab)))\n",
        "print(\"Sample tokens: {}\".format(random.sample(snt_vocab, 10)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Learnt vocab size: 31743\n",
            "Sample tokens: ['▁cafe', '▁slippery', 'xious', '▁resonate', '▁terrier', '▁feat', '▁frequencies', 'ainty', '▁punning', 'modern']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lJPtxrtz5470",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3YjqHVRpmlKq",
        "colab_type": "text"
      },
      "source": [
        "As we may observe, SentencePiece does quite the opposite to WordPiece. From the [documentation](https://github.com/google/sentencepiece/blob/master/README.md):\n",
        "\n",
        "\n",
        "SentencePiece first escapes the whitespace with a meta-symbol \"▁\" (U+2581) as follows:\n",
        "\n",
        "`Hello▁World`.\n",
        "\n",
        "Then, this text is segmented into small pieces, for example:\n",
        "\n",
        "`[Hello] [▁Wor] [ld] [.]`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OD-28l_0p0PQ",
        "colab_type": "text"
      },
      "source": [
        "Subwords which occur after whitespace (which are also those that most words begin with) are prepended with '▁', while others are unchanged. This excludes subwords which only occur at the beginning of sentences and nowhere else. These cases should be quite rare, however. \n",
        "\n",
        "So, in order to obtain a vocabulary analogous to WordPiece, we need to perform a simple conversion, removing \"▁\" from the tokens that contain it and adding \"##\"  to the ones that don't."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5QJGFjzOMbfN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def parse_sentencepiece_token(token):\n",
        "    if token.startswith(\"▁\"):\n",
        "        return token[1:]\n",
        "    else:\n",
        "        return \"##\" + token"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "64dcVgD98S28",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "bert_vocab = list(map(parse_sentencepiece_token, snt_vocab))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mL9ZR3RN9IMA",
        "colab_type": "text"
      },
      "source": [
        "We also add some special control symbols which are required by the BERT architecture. By convention, we put those at the beginning of the vocabulary."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VdTlXDPL8cHE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "ctrl_symbols = [\"[PAD]\",\"[UNK]\",\"[CLS]\",\"[SEP]\",\"[MASK]\"]\n",
        "bert_vocab = ctrl_symbols + bert_vocab"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ry1jvEr9EIdd",
        "colab_type": "text"
      },
      "source": [
        "We also append some placeholder tokens to the vocabulary. Those are useful if one wishes to update the pre-trained model with new, task-specific tokens. \n",
        "\n",
        "In that case, the placeholder tokens are replaced with new real ones, the pre-training data is re-generated, and the model is fine-tuned on new data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XLYSTil4E0Dm",
        "colab_type": "code",
        "outputId": "58086a68-17c6-4b46-e329-99723c329262",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "bert_vocab += [\"[UNUSED_{}]\".format(i) for i in range(VOC_SIZE - len(bert_vocab))]\n",
        "print(len(bert_vocab))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "32000\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qJKk_7JI-MtW",
        "colab_type": "text"
      },
      "source": [
        "Finally, we write the obtained vocabulary to file."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8G1jg0cj9Duf",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "VOC_FNAME = \"vocab.txt\" #@param {type:\"string\"}\n",
        "\n",
        "with open(VOC_FNAME, \"w\") as fo:\n",
        "  for token in bert_vocab:\n",
        "    fo.write(token+\"\\n\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "u5fs7H049nB8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1MqXZnc3FCuY",
        "colab_type": "text"
      },
      "source": [
        "Now let's see how the new vocabulary works in practice:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xsSOnEnC-jG1",
        "colab_type": "code",
        "outputId": "da87835b-bedd-4044-dbd2-713b48f2a0fb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 238
        }
      },
      "source": [
        "bert_tokenizer = tokenization.FullTokenizer(VOC_FNAME)\n",
        "bert_tokenizer.tokenize(testcase)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['color',\n",
              " '##less',\n",
              " 'geo',\n",
              " '##ther',\n",
              " '##mal',\n",
              " 'subs',\n",
              " '##tation',\n",
              " '##s',\n",
              " 'are',\n",
              " 'generat',\n",
              " '##ing',\n",
              " 'furious',\n",
              " '##ly']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RN8xDNfF0Q2q",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DauD8ndhEA-z",
        "colab_type": "text"
      },
      "source": [
        "Looking good!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EwFtStCo__QX",
        "colab_type": "text"
      },
      "source": [
        "## Step 5: generating pre-training data\n",
        "\n",
        "With the vocabulary at hand, we are ready to generate pre-training data for the BERT model. Since our dataset might be quite large, we will split it into shards:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xyN1nI04-uKV",
        "colab_type": "code",
        "outputId": "38d06556-73a6-4c93-82a2-acc8a6e8f94c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "!mkdir ./shards\n",
        "!split -a 4 -l 256000 -d $PRC_DATA_FPATH ./shards/shard_\n",
        "!ls ./shards/"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "shard_0000  shard_0001\tshard_0002  shard_0003\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w-FSq3zNFvLs",
        "colab_type": "text"
      },
      "source": [
        "Before we start generating, we need to set some model-specific parameters.  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UnZcD0yIBGPd",
        "colab_type": "code",
        "cellView": "both",
        "colab": {}
      },
      "source": [
        "MAX_SEQ_LENGTH = 128 #@param {type:\"integer\"}\n",
        "MASKED_LM_PROB = 0.15 #@param\n",
        "MAX_PREDICTIONS = 20 #@param {type:\"integer\"}\n",
        "DO_LOWER_CASE = True #@param {type:\"boolean\"}\n",
        "PROCESSES = 2 #@param {type:\"integer\"}\n",
        "PRETRAINING_DIR = \"pretraining_data\" #@param {type:\"string\"}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k-MibOBkFam2",
        "colab_type": "text"
      },
      "source": [
        "Now, for each shard we need to call *create_pretraining_data.py* script. To that end, we will employ the  *xargs* command. \n",
        "\n",
        "Running this might take quite some time depending on the size of your dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YbZjIeVP0T36",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "XARGS_CMD = (\"ls ./shards/ | \"\n",
        "             \"xargs -n 1 -P {} -I{} \"\n",
        "             \"python3 bert/create_pretraining_data.py \"\n",
        "             \"--input_file=./shards/{} \"\n",
        "             \"--output_file={}/{}.tfrecord \"\n",
        "             \"--vocab_file={} \"\n",
        "             \"--do_lower_case={} \"\n",
        "             \"--max_predictions_per_seq={} \"\n",
        "             \"--max_seq_length={} \"\n",
        "             \"--masked_lm_prob={} \"\n",
        "             \"--random_seed=34 \"\n",
        "             \"--dupe_factor=5\")\n",
        "\n",
        "XARGS_CMD = XARGS_CMD.format(PROCESSES, '{}', '{}', PRETRAINING_DIR, '{}', \n",
        "                             VOC_FNAME, DO_LOWER_CASE, \n",
        "                             MAX_PREDICTIONS, MAX_SEQ_LENGTH, MASKED_LM_PROB)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Fyo9_LcQ0pla",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "u5XzaY8xCdiV",
        "colab_type": "code",
        "outputId": "e5989b9c-d3ba-4cdf-c13f-01df6ed53829",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 12617
        }
      },
      "source": [
        "tf.gfile.MkDir(PRETRAINING_DIR)\n",
        "!$XARGS_CMD"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Reading from input files ***\n",
            "INFO:tensorflow:*** Reading from input files ***\n",
            "INFO:tensorflow:  ./shards/shard_0001\n",
            "INFO:tensorflow:  ./shards/shard_0000\n",
            "INFO:tensorflow:*** Writing to output files ***\n",
            "INFO:tensorflow:  pretraining_data/shard_0000.tfrecord\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] no it [MASK] her birthday [SEP] you [MASK] sung happy birthday to 20 women [MASK] [SEP]\n",
            "INFO:tensorflow:input_ids: 2 23 11 4 68 735 3 5 4 10213 294 735 9 577 449 4 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 3 8 15 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 8 73 287 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 0\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] [MASK] know what they say about a kid in short pants short pants [MASK] dick fuck off my thumbs are twice as big as yours bullshit who s got the biggest dick now runt you do ##ultures i am ii [MASK] nobody s bigger graphics [MASK] ooh renato s [MASK] school shh you come here [MASK] need some cigarettes [MASK] cigarettes do you want maced ##onia extra [SEP] malcolm kareem can [MASK] hear me [SEP]\n",
            "INFO:tensorflow:input_ids: 2 4 36 16 49 96 62 10 383 19 684 813 684 813 4 936 275 125 25 2326 38 1066 83 148 83 506 1189 75 8 63 7 1851 936 65 2443 5 31 15370 6 139 1717 4 425 8 1090 8514 4 616 1560 8 4 361 977 5 58 45 4 128 102 2508 4 2508 31 5 72 6590 6866 1406 3 1275 4512 34 4 240 18 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 1 14 37 41 45 46 50 56 60 67 72 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 5 684 23 1947 197 18 6247 6 16 1406 5 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] what a wonderful creature you are my love oh my love you ve taken my heart your beauty overwhelm ##s [MASK] l m spellbound [MASK] love for the first time [MASK] love come touch my heart [SEP] yamamura sadako will take the part that s all hey mr shigemori why her ignore them ok [MASK] hello hello who is this beige [MASK] you never speak andrei anna is [MASK] you anna do [MASK] [MASK] what will happen [SEP]\n",
            "INFO:tensorflow:input_ids: 2 16 10 598 2889 5 38 25 113 56 25 113 5 73 741 25 331 27 863 9260 57 4 233 26 21523 4 113 28 7 175 92 4 113 58 552 25 331 3 4578 792 79 94 7 399 14 8 40 95 209 3035 78 68 4208 105 192 4 212 212 75 17 22 15603 4 5 117 523 7707 848 17 4 5 848 31 4 4 16 79 436 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 5 11 21 25 31 32 55 61 62 69 73 74 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 5 113 18 25 25 113 1466 78 31 14 5 36 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] can t save up more money [MASK] s over now gene doesn t even turn 1 3 until [MASK] christmas it was 2 [UNK] and 1 0 cents in the tin i can t count it now but i trust you and [MASK] s 30 i [MASK] [MASK] which makes it 2 [UNK] and 1 0 cents it s not enough but it ll have to do selma bye bill selma selma stop i [MASK] [MASK] my gun [MASK] [MASK] selma i [MASK] t believe you [MASK] re [MASK] [MASK] to scare me i can [MASK] see a gun selma just [MASK] this feel [MASK] just feel it [SEP] feel it do you believe i have the gun i believe you [MASK] it s my money [SEP]\n",
            "INFO:tensorflow:input_ids: 2 34 12 497 51 129 188 4 8 120 65 802 208 12 142 289 204 437 412 4 1341 11 35 346 1 13 204 573 3023 19 7 14377 6 34 12 861 11 65 42 6 493 5 13 4 8 681 6 4 4 273 472 11 346 1 13 204 573 3023 11 8 39 235 42 11 47 29 9 31 397 347 629 397 397 168 6 4 4 25 461 4 4 397 6 4 12 223 5 4 33 4 4 9 1479 18 6 34 4 77 10 461 397 46 4 22 199 4 46 199 11 3 199 11 31 5 223 6 29 7 461 6 223 5 4 11 8 25 188 3\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:masked_lm_positions: 7 19 33 43 45 47 48 75 76 79 80 83 87 89 90 96 102 105 122 0\n",
            "INFO:tensorflow:masked_lm_ids: 14 202 34 11 681 63 287 26 6864 64 5 30 5 46 329 12 199 22 42 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 0\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] the back the seal the pyramid it [MASK] unfinished with the eye of god looking over it and the words [MASK] god favors our undertaking the seal is meant to be unfinished because this country is we [MASK] meant to keep [MASK] [MASK] [MASK] keep debating we re meant to read books [MASK] great scholars and talk about them which is why [MASK] [MASK] my name to a [MASK] cover i wanna be [MASK] assistant attorney general for [SEP] no it s more than one man who died in the last 12 years everyone who was there has died not one [MASK] [MASK] reporters at that demonstration is alive today [MASK] [MASK] have any idea [MASK] dr lkuma might be no 0 ##r his daughter sadako [SEP]\n",
            "INFO:tensorflow:input_ids: 2 7 85 7 2549 7 18160 11 4 14806 41 7 671 15 137 280 120 11 13 7 713 4 137 12717 114 14869 7 2549 17 896 9 37 14806 134 22 692 17 20 4 896 9 179 4 4 4 179 19917 20 33 896 9 447 925 4 167 7668 13 172 62 105 273 17 78 4 4 25 187 9 10 4 947 6 243 37 4 2813 3084 1549 28 3 23 11 8 129 197 61 89 75 575 19 7 200 905 213 405 75 35 50 126 575 39 61 4 4 7734 64 14 6549 17 591 287 4 4 29 152 328 4 473 5677 272 37 23 573 440 87 515 792 3\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:masked_lm_positions: 8 21 38 42 43 44 51 53 57 63 64 69 74 102 103 111 112 116 119 0\n",
            "INFO:tensorflow:masked_lm_ids: 8 21 33 145 182 9 447 119 172 6 6729 2946 27 15 7 31 5 103 272 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] 0 ##ne swimming class they went to the beach sadako was simply terrified lf you go in the sea you ll all [MASK] she said [SEP] i never said [MASK] [MASK] didn t say [MASK] [MASK] [MASK] [MASK] it it was just a joke were ##n t it faith she [MASK] [MASK] all right dad come and [MASK] down she ll be all right [SEP]\n",
            "INFO:tensorflow:input_ids: 2 573 2334 3750 881 49 250 9 7 1195 792 35 1499 4246 1800 5 52 19 7 778 5 47 40 4 53 135 3 6 117 135 4 4 112 12 96 4 4 4 4 11 11 35 46 10 1029 101 88 12 11 1200 53 4 4 40 54 216 58 13 4 115 53 47 37 40 54 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 23 30 31 35 36 37 38 51 52 58 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 307 6 6 6 112 12 48 47 37 395 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] i need [MASK] help [MASK] [MASK] find mr [MASK] in the morning i ll confess no i [MASK] t let them torment [MASK] no it s the only way your hand is hot [SEP] for a while billionaire ##k time for tv [MASK] you cold i ll [MASK] [MASK] a [MASK] of water you wait here is that the [MASK] of [MASK] go get [MASK] sadako go to the dressing room quick where s shigemori he s vanished what do we do we can t cancel we [MASK] go on mr toyama when [MASK] is [MASK] let s go away somewhere don t worry you ll be great you re beautiful we re going to open the doors it s all [MASK] the [MASK] touched him [SEP]\n",
            "INFO:tensorflow:input_ids: 2 6 128 4 153 4 4 164 209 4 19 7 310 6 47 2918 23 6 4 12 70 105 6516 4 23 11 8 7 130 118 27 371 17 478 3 28 10 327 7332 786 92 28 593 4 5 620 6 47 4 4 10 4 15 333 5 150 45 17 14 7 4 15 4 52 44 4 792 52 9 7 4845 301 888 103 8 3035 21 8 8678 16 31 20 31 20 34 12 2461 20 4 52 24 209 2067 90 4 17 4 70 8 52 163 756 30 12 351 5 47 37 167 5 33 365 20 33 106 9 326 7 2389 11 8 40 4 7 4 2433 60 3\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:masked_lm_positions: 3 5 6 9 18 23 38 43 48 49 51 60 62 65 88 94 96 122 124 0\n",
            "INFO:tensorflow:masked_lm_ids: 27 49 47 3035 158 5 573 38 44 5 464 362 11 725 47 22 120 54 231 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 0\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] [MASK] [MASK] full of ##hate my brother java foreign guy [SEP] will you sign youryearbook picture [SEP]\n",
            "INFO:tensorflow:input_ids: 2 4 4 481 15 21046 25 286 13126 3071 203 3 79 5 712 21700 696 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 1 2 8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 6 139 95 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 0\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] i know i am [MASK] loves you bill [MASK] verses [MASK] don [MASK] cry i shouldn t have told you don t worry [MASK] s ok i [MASK] t have no [MASK] think you ve got enough stuff would it make you feel better if i told you a secret what [MASK] you tell [MASK] [MASK] m [MASK] blind not yet but soon maybe sometime this [MASK] blind it s not as [SEP] it s a it s a family thing but blind [MASK] ve always known it from from [MASK] was a little [MASK] i knew sniffs and you re ok well i came to america [MASK] in [MASK] they can give gene an operation you know gene but [MASK] doesn t know about it [SEP]\n",
            "INFO:tensorflow:input_ids: 2 6 36 6 139 4 1000 5 629 4 20309 4 30 4 961 6 564 12 29 196 5 30 12 351 4 8 192 6 4 12 29 23 4 74 5 73 63 235 376 107 11 122 5 199 182 59 6 196 5 10 749 16 4 5 104 4 4 26 4 1040 39 411 42 433 155 2575 22 4 1040 11 8 39 83 3 11 8 10 11 8 10 258 143 42 1040 4 73 184 930 11 81 81 4 35 10 111 4 6 313 5892 13 5 33 192 69 6 236 9 970 4 19 4 49 34 131 802 98 1112 5 36 802 42 4 208 12 36 62 11 3\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:masked_lm_positions: 5 9 10 11 13 24 28 32 52 55 56 58 67 84 91 95 108 110 121 0\n",
            "INFO:tensorflow:masked_lm_ids: 53 629 6848 477 12 11 564 6 93 18 6 106 422 6 6 231 134 970 21 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 0\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] i don t know who i m up against [MASK] the deception don t try to hide [SEP] my [MASK] is boris and this is my family [MASK] sweden is samuel at [MASK] no [MASK] s not behaviour home he had to [MASK] on a little errand for the lord no [SEP]\n",
            "INFO:tensorflow:input_ids: 2 6 30 12 36 75 6 26 51 603 4 7 3682 30 12 247 9 794 3 25 4 17 2226 13 22 17 25 258 4 9566 17 2878 64 4 23 4 8 39 4623 174 21 97 9 4 24 10 111 15345 28 7 630 23 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 10 15 20 28 33 35 38 43 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 168 247 187 81 174 21 64 218 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] wait a minute you re saying debra got ray a dvd player because she wants something from him of course where you been wait that s so [MASK] [MASK] that s not her that s you [MASK] okay [SEP] did you ever think [MASK] that [SEP]\n",
            "INFO:tensorflow:input_ids: 2 150 10 457 5 33 341 3206 63 658 10 4405 2808 134 53 401 116 81 60 15 268 103 5 109 150 14 8 43 4 4 14 8 39 68 14 8 5 4 91 3 76 5 225 74 4 14 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 5 28 29 31 37 43 44 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 33 20 24788 8 84 74 62 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] this is summit team annie garrett [MASK] that you it s elliot [MASK] [MASK] s this mont ##gomer ##y wick how many are you three [MASK] different [MASK] i m with peter garrett i thought [MASK] only climbed solo rescue missions are appreciated what s your jonnie west ridge 23 000 you ##trained have to climb faster tell garrett annie bricks edema [SEP] listen come with me rhythm man [SEP]\n",
            "INFO:tensorflow:input_ids: 2 22 17 2396 546 844 1782 4 14 5 11 8 2695 4 4 8 22 3643 3858 221 2011 67 302 38 5 222 4 480 4 6 26 41 799 1782 6 171 4 130 2671 5554 1125 7557 38 10872 16 8 27 19786 994 4830 2863 534 5 21058 29 9 1217 991 104 1782 844 24689 3935 3 229 58 41 18 4079 89 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 7 13 14 26 28 36 40 43 47 53 61 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 17 1576 75 1916 10522 5 1125 480 2915 47 126 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] not exactly [MASK] [MASK] lovely you like [MASK] of course this couch is just like mine it s not like yours it is [MASK] in punctual bedroom you ll find your clothes your perfumes what are you playing at that s not all look [MASK] the view you like it you particle gone [MASK] far i m going [SEP] his name luis miranda ##dles [MASK] he lives at [UNK] si ##erra de [MASK] ##a he works on monte ##s ura ##les street real blue collar huh all [MASK] make it look like [MASK] robbery no people [MASK] trouble of [MASK] brother no people [MASK] trouble no shit all those who think that [MASK] s unfair need to know that it is ##n t so that life [SEP]\n",
            "INFO:tensorflow:input_ids: 2 39 441 4 4 1003 5 48 4 15 268 22 3675 17 46 48 367 11 8 39 48 506 11 17 4 19 21697 2616 5 47 164 27 1015 27 10173 16 38 5 716 64 14 8 39 40 82 4 7 1712 5 48 11 5 18166 330 4 470 6 26 106 3 87 187 4951 4513 28660 4 21 652 64 1 5023 9922 800 4 322 21 853 24 19262 57 31530 8949 664 256 706 3921 227 40 4 122 11 82 48 4 2820 23 138 4 610 15 4 286 23 138 4 610 23 176 40 181 75 74 14 4 8 5185 128 9 36 14 11 17 88 12 43 14 146 3\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:masked_lm_positions: 3 4 8 24 26 45 52 54 64 65 73 79 88 93 97 100 104 113 122 0\n",
            "INFO:tensorflow:masked_lm_ids: 11 8 11 506 7 64 73 121 2968 579 13259 57 54 10 23 268 23 146 88 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] nothing s over if there s hope [MASK] something i should have known you [MASK] come here and i did that [MASK] why i came we re through you re evil [MASK] [MASK] want to see you again help us for old times sake he s lost in the master in pile [SEP] now i [MASK] if i wanted a boyfriend it would be you jeff i just don t want one [MASK] now no problem you know on the back to [MASK] the bike there yeah bye jeff i [MASK] t think it s very [MASK] riding the bike wearing glasses and all [MASK] bye jeff jeff are you ok yeah [MASK] [MASK] fine hi [MASK] hi [MASK] heading home pregnant yes wanna throw that [SEP]\n",
            "INFO:tensorflow:input_ids: 2 166 8 120 59 50 8 375 4 116 6 133 29 930 5 4 58 45 13 6 76 14 4 78 6 236 20 33 262 5 33 1094 4 4 72 9 77 5 157 153 99 28 186 514 1107 21 8 305 19 7 1167 19 4550 3 65 6 4 59 6 246 10 1098 11 107 37 5 1059 6 46 30 12 72 61 4 65 23 291 5 36 24 7 85 9 4 7 1294 50 66 347 1059 6 4 12 74 11 8 124 4 1973 7 1294 1216 1371 13 40 4 347 1059 1059 38 5 192 66 4 4 230 217 4 217 4 1786 174 1828 84 243 828 14 3\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:masked_lm_positions: 8 15 22 32 33 56 60 73 83 85 91 97 105 109 113 114 117 119 122 0\n",
            "INFO:tensorflow:masked_lm_ids: 305 86 8 6 117 36 10 54 29 1294 30 589 397 38 6 26 397 629 56 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] but why he has forest to do ##ukka [MASK] come [MASK] i housewife not [MASK] going [MASK] [MASK] to that i just want to get out of here come on ann sofie no i m not so sure jacob we can t [MASK] tomas here ingar is probably dead by now now we re going to leave [MASK] else in this [MASK] maybe she s [MASK] dead let s do this spinach morning when we wake up all [MASK] of us ##rumbled this goddamn ##ed forest and no more fucking suggestions now let s go home thank you ok short ##est [MASK] takes first watch please can t you just do what i m [SEP] ok it s [MASK] no [MASK] can take the first watch [SEP]\n",
            "INFO:tensorflow:input_ids: 2 42 78 21 126 3004 9 31 19462 4 58 4 6 7352 39 4 106 4 4 9 14 6 46 72 9 44 55 15 45 58 24 2589 9105 23 6 26 39 43 149 3195 20 34 12 4 5498 45 7101 17 429 244 119 65 65 20 33 106 9 218 4 274 19 22 4 155 53 8 4 244 70 8 31 22 12198 310 90 20 675 51 40 4 15 99 13971 22 648 161 3004 13 23 129 283 8543 65 70 8 52 174 147 5 192 684 986 4 733 175 324 141 34 12 5 46 31 16 6 26 3 192 11 8 4 23 4 34 94 7 175 324 3\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:masked_lm_positions: 5 8 9 11 13 15 17 18 43 58 62 66 72 79 82 102 110 119 121 0\n",
            "INFO:tensorflow:masked_lm_ids: 166 41 22 24 26 142 9 4369 218 249 17970 39 407 393 218 745 46 18 6 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 0\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] intonation are you staring at [SEP] [MASK] s more than one man who died in [MASK] last 12 years everyone who was there has [MASK] not one of the [MASK] at that demonstration is alive wholeheartedly do you have any idea where dr lkuma [MASK] be no 0 ##r his daughter sadako i don t know so we don t even know yet when the funeral ##cooperative be [SEP]\n",
            "INFO:tensorflow:input_ids: 2 21703 38 5 2291 64 3 4 8 129 197 61 89 75 575 19 4 200 905 213 405 75 35 50 126 4 39 61 15 7 4 64 14 6549 17 591 15419 31 5 29 152 328 103 473 5677 4 37 23 573 440 87 515 792 6 30 12 36 43 20 30 12 142 36 411 90 7 2056 29824 37 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 1 7 16 20 25 30 36 45 63 67 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 16 11 7 405 575 7734 287 272 411 79 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] thank you [MASK] what s all the [MASK] [SEP] a vampire in the lobby did i hear that right never a dull [MASK] we re done get out now can you believe this it s a sweep [MASK] ll never make [MASK] [MASK] what is it [MASK] readers shouldn [MASK] we have been memo ##ed [MASK] s [MASK] [MASK] re [MASK] random mind readings [MASK] s lil ##ah it looks like i won t be able [MASK] make the 1 00 great overboard [MASK] to say this is a shame it s just a shame whenever i hear of disloyal ##ty it hurts me saying [MASK] personally but this sort of thing must be deal ##t with quickly and cleanly and una ##mb ##ig ##uous ##ly [SEP]\n",
            "INFO:tensorflow:input_ids: 2 147 5 4 16 8 40 7 4 3 10 4085 19 7 5693 76 6 240 14 54 117 10 5499 4 20 33 277 44 55 65 34 5 223 22 11 8 10 5278 4 47 117 122 4 4 16 17 11 4 14375 564 4 20 29 109 7002 161 4 8 4 4 33 4 7189 300 7573 4 8 9672 4110 11 308 48 6 158 12 37 686 4 122 7 204 614 167 6257 4 9 96 22 17 10 1431 11 8 46 10 1431 2144 6 240 15 15949 4648 11 2073 18 341 4 2278 42 22 669 15 143 191 37 421 180 41 1724 13 22141 13 22820 20188 14467 15735 353 3\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:masked_lm_positions: 3 8 23 38 42 43 47 50 56 58 59 61 65 77 83 84 105 106 118 0\n",
            "INFO:tensorflow:masked_lm_ids: 6957 4717 532 6 25 1065 300 12 14 78 49 343 11 9 6 29 2073 18 1724 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 0\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] bim bim ##mer who [MASK] [MASK] nan s zi ##mmer it [MASK] t matter now she s [MASK] stone slim [MASK] [MASK] the aged wes side in the house cos [MASK] day they were just like you keep off the crack keep off the crack help the aged respect your nan [MASK] ##cake day [MASK] ll [MASK] [MASK] too shake ya batty shake ya batty help the aged jarvis the famous batty shaking the batty help [MASK] aged help yourself ##iced the aged jarvis big im up fank ##s for watchin today i ope you ave come away with [MASK] [SEP] tim my man it sucks not being part of lalapalabala yeah hey dudes skyler what are you doing [MASK] [SEP]\n",
            "INFO:tensorflow:input_ids: 2 6744 6744 13422 75 4 4 6979 8 12404 22507 11 4 12 325 65 53 8 4 2038 4383 4 4 7 5209 29717 555 19 7 266 1138 4 160 49 101 46 48 5 179 125 7 2229 179 125 7 2229 153 7 5209 1128 27 6979 4 6281 160 4 47 4 4 121 1354 582 8300 1354 582 8300 153 7 5209 11735 7 1569 8300 4100 7 8300 153 4 5209 153 259 22822 7 5209 11735 148 5351 51 22869 57 28 5998 287 6 14400 5 22434 58 163 41 4 3 4807 25 89 11 2553 39 293 399 15 26063 66 95 20659 16357 16 38 5 145 4 3 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 4 5 6 12 18 21 22 31 40 52 53 55 57 58 77 81 100 120 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 75 839 18 30 222 13422 153 61 7 1138 61 5 37 1902 7 153 7922 45 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] but that s part [MASK] the challenge don [MASK] you [MASK] tired of doing [MASK] yes [MASK] [SEP] who s that in the kimono fraud it s nothing but a vulgar magic show dr lkuma you re being [MASK] mother what s [MASK] [SEP]\n",
            "INFO:tensorflow:input_ids: 2 42 14 8 399 4 7 2724 30 4 5 4 906 15 145 4 84 4 3 75 8 14 19 7 12599 3164 11 8 166 42 10 4570 1166 257 473 5677 5 33 293 4 296 16 8 4 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 5 9 11 15 17 39 43 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 15 12 44 1193 84 7511 254 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] [MASK] let s go to chil ##aquil s [MASK] faster get down man step on it the window [SEP] the [MASK] that s it [MASK] and [MASK] are you ready [MASK] s your stake ten thousand [MASK] [MASK] 10 000 go on then you re on for 10 10 s the stake heat em up [SEP]\n",
            "INFO:tensorflow:input_ids: 2 4 70 8 52 9 10478 9030 8 4 991 44 115 89 795 24 11 7 931 3 7 4 14 8 11 4 13 4 38 5 345 4 8 27 2753 597 1465 4 4 554 534 52 24 110 5 33 24 28 554 554 8 7 2753 1718 428 51 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 1 9 21 25 27 31 37 38 52 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 486 935 931 2016 1884 16 192 28 2753 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 0\n",
            "INFO:tensorflow:*** Writing to output files ***\n",
            "INFO:tensorflow:  pretraining_data/shard_0001.tfrecord\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] s [MASK] what has happened vissing call an ambulance now vissing wake up it s bjarne vissing bi there is an ambulance on its way his [MASK] [MASK] jesper [MASK] is your son arthur knew that jesper is [MASK] [MASK] that s why it was arthur who [UNUSED_222] her gotcha morphine it was of course you wrote the story about the drug addicts in ny [MASK] ##n [MASK] [UNK] [MASK] [MASK] for [MASK] woman you hadn t seen in fifteen years she was going [MASK] talk about the drug addicts those [SEP] about arthur you were to meet at a hotel in lt col ##bjorn ##sens ##gade but arthur beat you to it they [MASK] her dead of an overdose [MASK] woman was jesper ##sfer mother [SEP]\n",
            "INFO:tensorflow:input_ids: 2 8 4 16 126 260 14800 165 98 2494 65 14800 675 51 11 8 7926 14800 2784 50 17 98 2494 24 773 118 87 4 4 17234 4 17 27 263 5984 313 14 17234 17 4 4 14 8 78 11 35 5984 75 31970 68 9384 5934 11 35 15 268 5 1243 7 542 62 7 1928 9926 19 22596 4 88 4 1 4 4 28 4 270 5 1715 12 314 19 2995 213 53 35 106 4 172 62 7 1928 9926 181 3 62 5984 5 101 9 363 64 10 1044 19 543 6871 23420 30235 18680 42 5984 743 5 9 11 49 4 68 244 15 98 17489 4 270 35 17234 20668 296 3\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:masked_lm_positions: 2 27 28 30 39 40 48 49 50 58 66 68 70 71 73 85 115 121 125 0\n",
            "INFO:tensorflow:masked_lm_ids: 7926 187 17 21 27 263 435 68 7 7 25859 19 5 5397 10 9 312 14 8 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 0\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] [MASK] don [MASK] worry it s nothing embarrassing not anything like baton twirl ##ing or anything so what s your talent oh ##weird i [MASK] sorry twirl ##ing can be a [MASK] art ##protective saw this cheerleader [MASK] it at a football game she lit [MASK] [MASK] on fire and did this sexy dance i [MASK] i could do something like that why can t [MASK] my parents don t like anything oste ##nta ##tious [SEP] and they really don t like fire cheryl i think you have as good a chance as anyone to win you [MASK] in yourself [MASK] have gotten this far right really [MASK] [MASK] re so nice and [MASK] smart [MASK] so sensitive you re definitely gonna win that s it [SEP]\n",
            "INFO:tensorflow:input_ids: 2 4 30 4 351 11 8 166 2762 39 162 48 8823 5697 123 100 162 43 16 8 27 1419 56 24923 6 4 136 5697 123 34 37 10 4 940 19418 311 22 6312 4 11 64 10 3036 600 53 3007 4 4 24 558 13 76 22 1648 659 6 4 6 93 31 116 48 14 78 34 12 4 25 788 30 12 48 162 26973 27362 19055 3 13 49 108 30 12 48 558 1765 6 74 5 29 83 71 10 482 83 403 9 791 5 4 19 259 4 29 1265 22 470 54 108 4 4 33 43 194 13 4 719 4 43 2844 5 33 910 80 791 14 8 11 3\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:masked_lm_positions: 1 3 23 25 32 33 34 38 46 47 56 66 77 98 101 108 109 114 116 0\n",
            "INFO:tensorflow:masked_lm_ids: 42 12 1765 26 256 940 6 145 68 7696 516 5 13 223 9 66 5 43 13 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 0\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] god i did [MASK] and if [MASK] anyone tries to hurt condense of my new friends i would [MASK] them out i would make them suffer so much they d wish they were never born and if they circumvent weezing would hunt them down thank you kathy a brief shining moment and then that mouth helps if you pull it out good luck ladies i wanted to rub the crown for luck frank [MASK] it to get it polished i ll [MASK] it by the [MASK] they announce the [MASK] my god [MASK] s the crown yes it is [MASK] can taste it now [MASK] no no not the [SEP] this in the finale positions yes wear [MASK] crown be the crown you [MASK] the crown [SEP]\n",
            "INFO:tensorflow:input_ids: 2 137 6 76 4 13 59 4 403 3563 9 439 16286 15 25 170 342 6 107 4 105 55 6 107 122 105 1834 43 140 49 86 516 49 101 117 976 13 59 49 23725 21559 107 2138 105 115 147 5 899 10 4664 4588 532 13 110 14 736 3692 59 5 626 11 55 71 518 550 6 246 9 2672 7 1476 28 518 966 4 11 9 44 11 10164 6 47 4 11 119 7 4 49 5929 7 4 25 137 4 8 7 1476 84 11 17 4 34 1236 11 65 4 23 23 39 7 3 22 19 7 4453 5160 84 690 4 1476 37 7 1476 5 4 7 1476 3\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:masked_lm_positions: 4 7 12 19 26 33 34 39 40 70 74 82 86 90 93 100 105 118 124 0\n",
            "INFO:tensorflow:masked_lm_ids: 11 403 61 94 1834 101 117 996 6 1476 366 29 92 1886 11 5 23 7 38 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 0\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] the word is desecrated [SEP] that s not right come on so instead [MASK] a 13 [MASK] roam it s [UNK] but it s still just [MASK] number [SEP]\n",
            "INFO:tensorflow:input_ids: 2 7 458 17 10681 3 14 8 39 54 58 24 43 1178 4 10 1708 4 10008 11 8 1 42 11 8 159 46 4 618 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 14 17 18 27 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 15 422 1085 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] i did that no [MASK] did that and [MASK] [MASK] of highly specialize ##d federal manicurists some people that make you you know like [MASK] you [MASK] what you are under [MASK] get in the car no you think you saved something tonight [MASK] all you did was [MASK] [MASK] the dream of young women all over this splen what you think their dream is to get blown up you really got a good shot at that [MASK] plea yeah well i earned it honey twenty five years [MASK] [MASK] ##g [MASK] queens and what do i get fired they steal [MASK] life they [MASK] my beauty pageant hey hey it is not a beauty pageant [MASK] is [MASK] scholarship program yeah [SEP] is it serious [SEP]\n",
            "INFO:tensorflow:input_ids: 2 6 76 14 23 4 76 14 13 4 4 15 2323 10585 206 2541 11721 102 138 14 122 5 5 36 48 4 5 4 16 5 38 423 4 44 19 7 201 23 5 74 5 1018 116 420 4 40 5 76 35 4 4 7 628 15 451 449 40 120 22 21990 16 5 74 193 628 17 9 44 2658 51 5 108 63 10 71 540 64 14 4 5592 66 69 6 3187 11 380 1334 354 213 4 4 1271 4 4341 13 16 31 6 44 1478 49 1160 4 146 49 4 25 863 964 95 95 11 17 39 10 863 964 4 17 4 2838 2010 66 3 17 11 581 3\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1\n",
            "INFO:tensorflow:masked_lm_positions: 5 9 10 25 27 32 44 49 50 59 78 89 90 92 93 102 105 117 119 0\n",
            "INFO:tensorflow:masked_lm_ids: 2584 10 546 365 36 1420 42 9 1440 692 5457 15 28709 863 4341 25 1160 11 10 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] i m just thinking this this [MASK] it s a [MASK] it doesn [MASK] suck are you boyzvoice i m as ##bjorn [MASK] organize ##r hi there [MASK] spoke [MASK] the [MASK] i see you brought a whole team we only [MASK] one rule here no punching old ladies in [MASK] we ll have none of that where are your vocal ##ists by [MASK] way can you believe the crap those newspapers print no [MASK] here will suspect you of playback thurm everything triumvirate live we [MASK] all our music on cd we need to play this you can t [SEP] [MASK] is live [MASK] what do you mean we don t have [SEP]\n",
            "INFO:tensorflow:input_ids: 2 6 26 46 427 22 22 4 11 8 10 4 11 208 4 1332 38 5 4286 6 26 83 23420 4 9360 440 217 50 4 1386 4 7 4 6 77 5 524 10 359 546 20 130 4 61 1705 45 23 19056 186 550 19 4 20 47 29 1010 15 14 103 38 27 11946 7720 119 4 118 34 5 223 7 1012 181 8752 3966 23 4 45 79 954 5 15 12321 31382 189 23411 316 20 4 40 114 584 24 7035 20 128 9 391 22 5 34 12 3 4 17 316 4 16 31 5 132 20 30 12 29 3 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 7 11 14 23 28 30 32 42 51 64 75 82 84 87 102 105 110 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 2553 111 12 7 20 24 443 29 2372 7 61 45 8 29 189 45 20 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 0\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] not this [SEP] you go [MASK] where am i gonna put my [MASK] [SEP]\n",
            "INFO:tensorflow:input_ids: 2 39 22 3 5 52 4 103 139 6 80 156 25 4 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 6 13 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 231 461 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] but this [MASK] this is [MASK] much better than [MASK] [MASK] [MASK] even better and better than very very best moment in my life [SEP] well when you choose to go it ll be ##your choice [MASK] on to they ##outh minister you need church mom why [MASK] doing this [SEP]\n",
            "INFO:tensorflow:input_ids: 2 42 22 4 22 17 4 140 182 197 4 4 4 142 182 13 182 197 124 124 253 532 19 25 146 3 69 90 5 1480 9 52 11 47 37 1802 1070 4 24 9 49 27523 5198 5 128 1078 298 78 4 145 22 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 3 6 10 11 12 37 41 48 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 22 124 593 11 8 241 27523 3252 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] oh god that was lapps [MASK] was a kiss with your wet gross tongue hanging out ooh is [MASK] how boys back east kiss that s how everybody kisses charlotte [SEP] good [MASK] good job i got [MASK] something i could ##n [MASK] [MASK] ate pizza you [MASK] panties you re wild we [MASK] on this remember [SEP]\n",
            "INFO:tensorflow:input_ids: 2 56 137 14 35 16493 4 35 10 729 41 27 2984 3219 2462 1609 55 616 17 4 67 549 85 1297 729 14 8 67 335 3918 4289 3 71 4 71 309 6 63 4 116 6 93 88 4 4 3291 1559 5 4 2677 5 33 1325 20 4 24 22 239 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 5 6 19 33 38 43 44 48 54 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 1414 11 14 309 5 12 5 839 847 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] can ##mtex believe it jason surrender ##ed himself to colchis and this [MASK] [MASK] is very beautiful actually she is very beautiful you fools they lost faith i do not think jason abandon us he [MASK] forfiet ##residency [MASK] what does she treated to [MASK] [SEP] what does she means to us [SEP]\n",
            "INFO:tensorflow:input_ids: 2 34 16944 223 11 2924 5289 161 693 9 15389 13 22 4 4 17 124 365 387 53 17 124 365 5 7399 49 305 1200 6 31 39 74 2924 2802 99 21 4 26317 15019 4 16 210 53 2628 9 4 3 16 210 53 553 9 99 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 2 13 14 36 38 39 43 45 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 5 1804 53 107 87 7833 553 60 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 0\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] [MASK] [SEP] what are you doing [SEP]\n",
            "INFO:tensorflow:input_ids: 2 4 3 16 38 5 145 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 149 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 0\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] george oh that cycles in particular we [MASK] together like this me and george that s what [MASK] me you re worried about [MASK] flying junkie you [MASK] mystery solver roll up roll up [SEP] you better solve this [MASK] the fuck could they release that album [MASK] me on the cover [SEP]\n",
            "INFO:tensorflow:input_ids: 2 1145 56 14 3289 19 3185 20 4 269 48 22 18 13 1145 14 8 16 4 18 5 33 1051 62 4 1502 5761 5 4 2761 11155 1182 51 1182 51 3 5 182 2258 22 4 7 275 93 49 1579 14 3618 4 18 24 7 947 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 4 8 18 24 28 40 41 48 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 3618 1607 3714 102 148 67 7 321 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 0\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] your application is a novel idea however it is [MASK] the role of [MASK] court to evaluate novel ideas and as such i [MASK] grant your [MASK] [MASK] honor ms [MASK] will not see justice [MASK] an arrest warrant [MASK] [MASK] are you going to appeal this decision yes [MASK] honor good because i ve already passed the paperwork [MASK] to an appell ##ate term judge in anticipation horrend stash appeal you have yes i have because [MASK] my decision is reversed i hope [MASK] s done soon enough to do you some good [SEP] this hearing is adjourned ms kraft you didn t [MASK] the rest of his decision it s going to [MASK] appealed it s going [MASK] be okay it s not [MASK] [SEP]\n",
            "INFO:tensorflow:input_ids: 2 27 5666 17 10 7895 328 2124 11 17 4 7 3240 15 4 776 9 10229 7895 2421 13 83 357 6 4 3656 27 4 4 912 7765 4 79 39 77 2662 4 98 1420 3817 4 4 38 5 106 9 8479 22 1331 84 4 912 71 134 6 73 323 1269 7 4819 4 9 98 22501 3220 3403 1397 19 13032 30044 6095 8479 5 29 84 6 29 134 4 25 1331 17 14937 6 375 4 8 277 433 235 9 31 5 102 71 3 22 2005 17 6029 7765 7172 5 112 12 4 7 499 15 87 1331 11 8 106 9 4 14897 11 8 106 4 37 91 11 8 39 4 3\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:masked_lm_positions: 10 14 24 27 28 31 36 40 41 50 60 69 70 78 85 105 115 120 126 0\n",
            "INFO:tensorflow:masked_lm_ids: 39 22 689 5666 27 7172 836 17 12778 27 24 15 98 59 11 240 37 9 91 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 0\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] incomplete yes and [MASK] said [SEP] jesus is just [MASK] right ##with me us jesus [MASK] [MASK] all right oh yeah jesus is just all right ##with me jesus [MASK] just all right ##with me okay let [MASK] take a look atyour pictures kitty what doyou see when you think ##ofgod [SEP]\n",
            "INFO:tensorflow:input_ids: 2 10758 84 13 4 135 3 454 17 46 4 54 3852 18 99 454 4 4 40 54 56 66 454 17 46 40 54 3852 18 454 4 46 40 54 3852 18 91 70 4 94 10 82 15107 1554 2708 16 5410 77 90 5 74 26929 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 1 4 10 16 17 30 38 49 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 8060 137 40 17 46 17 8 5 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] fucking strong i always look people in the eye when i [MASK] talking [MASK] them i don t make [MASK] unnecessary moves the first ten words are enough first five amigo [MASK] gestures and i know [MASK] in his psyche fear is lurking he s not [MASK] [MASK] the fact that he s just about to foul his agents [MASK] know why because his weak psyche just mailed his inch [MASK] turd saying there s [MASK] meeting down [MASK] trousers this is incredible i have to go [MASK] the john what do you mean i [MASK] get a immi [SEP] they r e still in it [SEP]\n",
            "INFO:tensorflow:input_ids: 2 283 742 6 184 82 138 19 7 671 90 6 4 271 4 105 6 30 12 122 4 6125 2687 7 175 597 713 38 235 175 354 15536 4 6733 13 6 36 4 19 87 8738 1136 17 10709 21 8 39 4 4 7 769 14 21 8 46 62 9 5980 87 1812 4 36 78 134 87 1996 8738 46 15197 87 2907 4 7607 341 50 8 4 849 115 4 5179 22 17 1451 6 29 9 52 4 7 563 16 31 5 132 6 4 44 10 22789 3 49 998 721 159 19 11 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 2 12 14 20 32 37 47 48 59 60 71 76 79 88 96 99 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 742 26 9 152 8 14 2269 15 813 5 123 10 87 9 133 5588 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] i think you might have been right frank is kathy morningside s [MASK] disgusting pervert ##ed frank he cleared under another name i ran a new cc [MASK] [MASK] [MASK] even a weapons charge [SEP] ##dogs we doing full deployment [MASK] didn t want to ##ser about [MASK] [SEP]\n",
            "INFO:tensorflow:input_ids: 2 6 74 5 272 29 109 54 966 17 899 2199 8 4 1414 2321 161 966 21 2856 423 255 187 6 996 10 170 24973 4 4 4 142 10 1517 1117 3 25089 20 145 481 11023 4 112 12 72 9 25386 62 4 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 13 28 29 30 36 41 46 48 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 263 578 19212 3705 38 2678 240 11 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 0\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] the one that said ##ema [MASK] name [SEP] now [SEP]\n",
            "INFO:tensorflow:input_ids: 2 7 61 14 135 18918 4 187 3 65 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 5 6 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 6342 8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] jean jesus something ble ##w this way here thanks and here [SEP] he had [MASK] the money and claimed to know your bodyguard he came to collect [MASK] assured conclusion maybe he was [MASK] witted [MASK] was witted enough to nick the case any idea whose this vol ##ks ##wag ##en is [MASK] have ##n t seen it [MASK] exactly it [MASK] been here for a couple of days now [MASK] m [MASK] [MASK] take a closer look hey bolec i have this weird feeling like [MASK] seeing this face the face of the guy who sent [MASK] the bullet it [MASK] really weird [MASK] a sip you know i remember just one thing [MASK] vividly about that evening yes that doll on the dancing pole [SEP]\n",
            "INFO:tensorflow:input_ids: 2 999 454 116 8206 2236 22 118 45 211 13 45 3 21 97 4 7 188 13 5817 9 36 27 6200 21 236 9 3530 4 6741 6198 155 21 35 4 11868 4 35 11868 235 9 3065 7 459 152 328 1668 22 8948 11118 31519 1287 17 4 29 88 12 314 11 4 441 11 4 109 45 28 10 604 15 360 65 4 26 4 4 94 10 1370 82 95 4763 6 29 22 821 571 48 4 855 22 332 7 332 15 7 203 75 866 4 7 2581 11 4 108 821 4 10 9441 5 36 6 239 46 61 143 4 10398 62 14 708 84 14 2003 24 7 1109 2362 3\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:masked_lm_positions: 15 18 28 29 34 36 46 53 59 62 71 73 74 87 98 102 105 115 117 0\n",
            "INFO:tensorflow:masked_lm_ids: 524 13 7 3369 9748 21 1668 6 195 8 6 106 9 4720 18 8 94 602 62 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] [MASK] lord he s prov ##ing this shit ain t gonna [MASK] us all what was that [MASK] said this [MASK] [MASK] t gonna kill us all somebody s always gonna see the other side you feeling it they [MASK] them pink tops are pretty good i went to see this [MASK] the one about [MASK] they [MASK] to the jews in the war you went to see a movie lord what they did to them people what ##oratory of dope fiend [MASK] to the goddamn movies the germans decided that they were ##n [MASK] human no more they just [SEP] i hate his guts but i hate farnsworth even more now that he s so hot that hot beard lurk [MASK] what [MASK] my beard [SEP]\n",
            "INFO:tensorflow:input_ids: 2 4 630 21 8 14598 123 22 176 382 12 80 4 99 40 16 35 14 4 135 22 4 4 12 80 228 99 40 364 8 184 80 77 7 183 555 5 571 11 49 4 105 1842 3562 38 350 71 6 250 9 77 22 4 7 61 62 4 49 4 9 7 2473 19 7 636 5 250 9 77 10 870 630 16 49 76 9 105 138 16 25361 15 4441 10592 4 9 7 648 1504 7 2134 921 14 49 101 88 4 605 23 129 49 46 3 6 551 87 3430 42 6 551 12545 142 129 65 14 21 8 43 478 14 478 5718 18189 4 16 4 25 5718 3\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:masked_lm_positions: 1 10 12 18 21 22 35 40 52 56 58 79 83 95 109 112 121 122 124 0\n",
            "INFO:tensorflow:masked_lm_ids: 84 12 228 6 176 382 555 96 870 16 76 251 460 12 60 65 161 2022 62 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] gentlemen one moment are peak going to [MASK] off my hand because of some fucking case how am [MASK] going to look like [SEP] i [MASK] t want to be crippled for [MASK] rest of my life i have a family holiday plans i am a businessman [MASK] need [MASK] afterward to work pushup my laptop and to scratch rubbers [MASK] i m sure i have the key in question somewhere here then start looking for it sugar now [MASK] well oh sorry you fucking [MASK] what have you fucking done to me you thug stop whining boys don t cry it had to [MASK] so wonderful this is the answering [MASK] of kuba brenner leave your message after [MASK] beep kuba i know you are [SEP]\n",
            "INFO:tensorflow:input_ids: 2 833 61 532 38 8793 106 9 4 125 25 371 134 15 102 283 459 67 139 4 106 9 82 48 3 6 4 12 72 9 37 12277 28 4 499 15 25 146 6 29 10 258 4850 1435 6 139 10 6270 4 128 4 15352 9 154 27840 25 8659 13 9 2486 10826 4 6 26 149 6 29 7 1035 19 512 756 45 110 306 280 28 11 1606 65 4 69 56 136 5 283 4 16 29 5 283 277 9 18 5 13964 168 4938 549 30 12 961 11 97 9 4 43 598 22 17 7 5140 4 15 1990 5242 218 27 1164 202 4 5259 1990 6 36 5 38 3\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:masked_lm_positions: 5 8 19 26 33 47 48 50 51 54 60 61 80 86 89 93 105 112 120 0\n",
            "INFO:tensorflow:masked_lm_ids: 5 2886 6 30 7 6270 6 22 371 24 25 303 124 936 5 18 37 809 7 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 0\n",
            "INFO:tensorflow:Wrote 91978 total instances\n",
            "INFO:tensorflow:*** Reading from input files ***\n",
            "INFO:tensorflow:  ./shards/shard_0002\n",
            "INFO:tensorflow:Wrote 104109 total instances\n",
            "INFO:tensorflow:*** Reading from input files ***\n",
            "INFO:tensorflow:  ./shards/shard_0003\n",
            "INFO:tensorflow:*** Writing to output files ***\n",
            "INFO:tensorflow:  pretraining_data/shard_0002.tfrecord\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] did you say something my baby [MASK] that you yeah [MASK] s me i love you i would never hurt you schem know ##eeeeeeeeeeeeeeee don t you i know you re my child you [MASK] to go you have to go [SEP] meeting ##in the office [MASK] [MASK] in the hall [SEP]\n",
            "INFO:tensorflow:input_ids: 2 76 5 96 116 25 238 4 14 5 66 4 8 18 6 113 5 6 107 117 439 5 12517 36 21029 30 12 5 6 36 5 33 25 498 5 4 9 52 5 29 9 52 3 849 494 7 694 4 4 19 7 2716 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 7 11 22 24 35 45 47 48 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 17 11 5 14 29 7 10 849 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] dog [MASK] mendoza sick can you believe that [MASK] [MASK] sick dog it [MASK] horrible my mother my [MASK] kept telling us how much fun we were having but we were ##n t having any fun my dad drove himself crazy trying to [MASK] sure we [MASK] fun it s okay [MASK] we know [MASK] you feel yeah dad we know exactly howyou feel barking oh [SEP] [MASK] honey i m okay [MASK] m all right beth at rad we re out ofthe bug zone honey richard [MASK] s it we re taking a night off ##from this [MASK] we re staying in a hotel brennan yes beth the dog [MASK] eaten up all our reserve cash we can t afford to stay in a hotel [SEP]\n",
            "INFO:tensorflow:input_ids: 2 304 4 18273 608 34 5 223 14 4 4 608 304 11 4 1983 25 296 25 4 1210 678 99 67 140 445 20 101 400 42 20 101 88 12 400 152 445 25 216 2239 693 414 329 9 4 149 20 4 445 11 8 91 4 20 36 4 5 199 66 216 20 36 441 23119 199 2641 56 3 4 380 6 26 91 4 26 40 54 4185 64 11147 20 33 55 2847 2962 3353 380 1516 4 8 11 20 33 509 10 169 125 9003 22 4 20 33 1580 19 10 1044 6117 84 4185 7 304 4 2479 51 40 114 6892 1529 20 34 12 2205 9 214 19 10 1044 3\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:masked_lm_positions: 2 3 9 10 14 19 23 39 42 44 47 52 55 68 73 79 88 99 111 0\n",
            "INFO:tensorflow:masked_lm_ids: 35 201 10 201 35 296 67 2239 329 122 97 431 67 823 6 502 14 1814 126 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 0\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] if we can condition the mind to prolong ##ed activity [SEP] we [MASK] halfway to creating a clone no at least give me a [MASK] to prove it [SEP]\n",
            "INFO:tensorflow:input_ids: 2 59 20 34 2644 7 300 9 15763 161 3684 3 20 4 6271 9 5063 10 739 23 64 502 131 18 10 4 9 1309 11 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 13 15 23 25 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 33 9 18 482 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 0\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] very important [MASK] s what [MASK] re looking [MASK] don who s behind this stan [SEP] while [MASK] was deliveri ##ng some merchandise in my escargot i suddenly found myself amids ##t a storm [SEP]\n",
            "INFO:tensorflow:input_ids: 2 124 430 4 8 16 4 33 280 4 30 75 8 528 22 1198 3 327 4 35 31476 1230 102 5871 19 25 13503 6 1317 312 377 28864 180 10 1396 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 3 6 9 18 21 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 14 20 28 6 1230 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] well manel what [MASK] it [MASK] was the [MASK] that finished me and the cold you ve got to fight [SEP] what s happening to [MASK] [SEP]\n",
            "INFO:tensorflow:input_ids: 2 69 4388 16 4 11 4 35 7 4 14 793 18 13 7 620 5 73 63 9 624 3 16 8 846 9 4 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 4 6 9 26 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 17 11 3783 4200 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] now [MASK] nolan this is top [MASK] we re after a hardened ##crafts what i need you to do [MASK] stake out the bank can you handle that no problem all right now what i m [MASK] is shut your [MASK] on this one i mean it hush don t say [MASK] na ##da [MASK] with me all right if you can do that we might find a place for [MASK] on [MASK] force you d be in in the bureau right in the bureau probably you d get the f but you won t [MASK] the bi until some years have gone [MASK] [SEP] right cos there s some work involved [MASK] [MASK] the bank work [MASK] m all over it [MASK] maintained damn trent [SEP]\n",
            "INFO:tensorflow:input_ids: 2 65 4 4476 22 17 633 4 20 33 202 10 7807 29588 16 6 128 5 9 31 4 2753 55 7 771 34 5 878 14 23 291 40 54 65 16 6 26 4 17 373 27 4 24 22 61 6 132 11 3977 30 12 96 4 1890 5399 4 41 18 40 54 59 5 34 31 14 20 272 164 10 224 28 4 24 4 868 5 86 37 19 19 7 5046 54 19 7 5046 429 5 86 44 7 574 42 5 158 12 4 7 2784 412 102 213 29 330 4 3 54 1138 50 8 102 154 1620 4 4 7 771 154 4 26 40 120 11 4 14312 290 1673 3\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:masked_lm_positions: 2 7 13 19 20 37 41 52 55 71 73 83 96 104 113 114 118 123 124 0\n",
            "INFO:tensorflow:masked_lm_ids: 229 749 2499 31 17 341 736 1203 5 5 7 19 44 119 441 48 6 347 478 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 0\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] and what s that got to do with fives it took [MASK] trained to close her up i [MASK] t like fives [SEP] that s [MASK] let s hit it let s go let s go hon will you take a [MASK] of [MASK] with my new camera wait [MASK] [MASK] let s [MASK] come on [MASK] on i m gonna miss you [SEP]\n",
            "INFO:tensorflow:input_ids: 2 13 16 8 14 63 9 31 41 7596 11 366 4 4273 9 463 68 51 6 4 12 48 7596 3 14 8 4 70 8 446 11 70 8 52 70 8 52 7600 79 5 94 10 4 15 4 41 25 170 1279 150 4 4 70 8 4 58 24 4 24 6 26 80 284 5 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 12 13 19 26 42 44 50 51 54 57 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 3677 3943 30 11 696 99 40 190 52 58 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 0\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] [MASK] order to kill mt holy and exterminate our tribe to do so they needed the seed of hwa ##san royal family soo the chief of mae tempted the king of haw ##san [MASK] and gave a birth to vee who s to complete the sword [MASK] it wasn t [MASK] [SEP] here to protect our tribe and mt holy if vee is here [MASK] protect us why sending her [MASK] [MASK] [UNUSED_101] when the black [MASK] appears in [MASK] few days it ll be the last [MASK] for mae to perfect the sword they ll [MASK] to get hold of vee and complete [MASK] holy sword so mt [MASK] [MASK] using her as the sacrifice it s the wisdom of mt holy [MASK] protect our [SEP]\n",
            "INFO:tensorflow:input_ids: 2 4 601 9 228 8234 734 13 11132 114 2270 9 31 43 49 812 7 9214 15 10138 19141 4018 258 24707 7 1500 15 2944 8004 7 974 15 31381 19141 4 13 435 10 2954 9 4154 75 8 9 1640 7 1677 4 11 282 12 4 3 45 9 1032 114 2270 13 8234 734 59 4154 17 45 4 1032 99 78 2783 68 4 4 31849 90 7 501 4 2752 19 4 398 360 11 47 37 7 200 4 28 2944 9 611 7 1677 49 47 4 9 44 252 15 4154 13 1640 4 734 1677 43 8234 4 4 1179 68 83 7 2627 11 8 7 4458 15 8234 734 4 1032 114 3\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:masked_lm_positions: 1 34 47 51 65 71 72 73 77 80 88 89 97 105 110 111 121 124 125 0\n",
            "INFO:tensorflow:masked_lm_ids: 19 6146 42 320 9 9 8234 734 1391 10 482 28 247 7 734 17 15 9 1032 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 0\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] let her go hey hey hey [SEP] s that left it [MASK] i know damn i ve been tricked it quote be done [MASK] those [MASK] i won t let [MASK] get [MASK] [MASK] s stone take the sacred stone [MASK] the mountains await further instructions there do you want me to do anything for ru bing [MASK] what master you like ru bing do you not [MASK] can give up my life for her i am touched by your [MASK] master everything [MASK] be back in order once we get [MASK] the [MASK] s [MASK] i will betroth ru bing to you then magnani ##mous of me is ##n t it love is [MASK] love does but the [MASK] stone is [UNUSED_227] [MASK] huan jen [SEP]\n",
            "INFO:tensorflow:input_ids: 2 70 68 52 95 95 95 3 8 14 237 11 4 6 36 290 6 73 109 5135 11 6509 37 277 4 181 4 6 158 12 70 4 44 4 4 8 2038 94 7 2456 2038 4 7 3520 8062 1510 8910 50 31 5 72 18 9 31 162 28 4916 5329 4 16 1167 5 48 4916 5329 31 5 39 4 34 131 51 25 146 28 68 6 139 2433 119 27 4 1167 189 4 37 85 19 601 348 20 44 4 7 4 8 4 6 79 16387 4916 5329 9 5 110 29283 30415 15 18 17 88 12 11 113 17 4 113 210 42 7 4 2038 17 31975 4 6840 3849 3\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:masked_lm_positions: 12 21 24 26 31 33 34 41 58 68 81 84 92 94 96 115 120 123 124 0\n",
            "INFO:tensorflow:masked_lm_ids: 50 191 119 7908 403 7 973 9 1167 6 113 79 85 973 2038 83 2456 65 41 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] you don t love [MASK] you don t love me at all sobs you [MASK] any better [MASK] y are i got toast oh thanks love what [MASK] you mum i [MASK] i wanted it off my [MASK] i delvian [MASK] re mad you know it it was nothin faith you won t [MASK] anything will you to your sisters mum if i was enrich say anything i d [MASK] said it years ago [MASK] [SEP] sorry oh what [MASK] i done mum what am i gonna do [MASK] has ##n t ever looked [MASK] me how he did in that park [MASK] ever that s cos all [MASK] s seein is [MASK] sex and nothing else and that s how men are i [MASK] sorry [SEP]\n",
            "INFO:tensorflow:input_ids: 2 5 30 12 113 4 5 30 12 113 18 64 40 6848 5 4 152 182 4 612 38 6 63 2400 56 211 113 16 4 5 1807 6 4 6 246 11 125 25 4 6 8078 4 33 885 5 36 11 11 35 1203 1200 5 158 12 4 162 79 5 9 27 2928 1807 59 6 35 21928 96 162 6 86 4 135 11 213 349 4 3 136 56 16 4 6 277 1807 16 139 6 80 31 4 126 88 12 225 766 4 18 67 21 76 19 14 1082 4 225 14 8 1138 40 4 8 22769 17 4 726 13 166 274 13 14 8 67 404 38 6 4 136 3\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:masked_lm_positions: 5 15 18 28 32 38 40 41 54 65 70 75 80 89 95 103 109 113 125 0\n",
            "INFO:tensorflow:masked_lm_ids: 18 3180 45 5321 246 1775 171 5 96 80 29 6 29 21 64 1200 21 7 26 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 0\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] advertisement in the supermarket red mountain ##bike missing [SEP] shortlisted for best laure tv illusionist groves ##nor house on the 23 ##rd amazing you wouldn t think that was possible same as a hurricane driving a [MASK] of grass into a tree [MASK] maybe they have graduate t made a decision [MASK] it for your after dinners it s in the wrist action with my luck [MASK] [MASK] take a shuffle toupee ##s off talking of wrist action jonathan ross called i agreed [MASK] [MASK] [MASK] talk show at a [MASK] bar mitz [MASK] last [MASK] i wonder if i should cancel the guy s so hip and [MASK] ##ext ##ual with the [MASK] last time [MASK] said two words and the whole place fell about [SEP]\n",
            "INFO:tensorflow:input_ids: 2 11390 19 7 11759 751 1491 28727 1412 3 21367 28 253 22365 593 21516 18813 19223 266 24 7 2863 3482 1084 5 318 12 74 14 35 901 281 83 10 3507 1186 10 4 15 3210 173 10 779 4 155 49 29 7447 12 226 10 1331 4 11 28 27 202 13896 11 8 19 7 6619 1242 41 25 518 4 4 94 10 15767 14827 57 125 271 15 6619 1242 2076 1360 343 6 1977 4 4 4 172 257 64 10 4 1302 18893 4 200 4 6 757 59 6 133 2461 7 203 8 43 3875 13 4 31257 13234 41 7 4 200 92 4 135 127 713 13 7 359 224 838 62 3\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:masked_lm_positions: 13 23 37 43 47 52 67 68 71 74 84 85 86 91 94 96 109 114 117 0\n",
            "INFO:tensorflow:masked_lm_ids: 2203 1084 4378 21367 88 340 6 86 398 125 9 31 87 11945 19078 507 31510 3317 21 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] apart from you that is we re just going [MASK] sleep next to each other aren t we nothing else [MASK] good good night [MASK] i [MASK] t understand it s damn strange [SEP] [MASK] can sit and laugh but this is serious barbara understand [SEP]\n",
            "INFO:tensorflow:input_ids: 2 1282 81 5 14 17 20 33 46 106 4 416 288 9 471 183 438 12 20 166 274 4 71 71 169 4 6 4 12 234 11 8 290 683 3 4 34 395 13 981 42 22 17 581 12182 234 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 10 15 21 25 27 35 44 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 9 183 23 2917 30 5 5 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] i ve been in bad patches so many times and came home with so much stock little [MASK] like [MASK] [MASK] [MASK] pack it on the pier i [MASK] find the fish always and i will this time so don t fuck [MASK] me aye aye skip put them closer murph closer give them a goddamn [MASK] aye aye cap you heard him boys boss ain t happy double time [SEP] all right let debbie [MASK] to get some sleep [MASK] s easy for you to say [MASK] [MASK] t think you d even [MASK] talk to me sherry and i m not a little girl any more and i know that you ve probably heard some things that i m [MASK] very [MASK] [MASK] well [SEP]\n",
            "INFO:tensorflow:input_ids: 2 6 73 109 19 215 6148 43 302 514 13 236 174 41 43 140 3286 111 4 48 4 4 4 1387 11 24 7 3181 6 4 164 7 596 184 13 6 79 22 92 43 30 12 275 4 18 1722 1722 775 156 105 1370 1185 1370 131 105 10 648 4 1722 1722 1108 5 319 60 549 782 382 12 294 867 92 3 40 54 70 1827 4 9 44 102 416 4 8 320 28 5 9 96 4 4 12 74 5 86 142 4 172 9 18 1187 13 6 26 39 10 111 231 152 129 13 6 36 14 5 73 429 319 102 185 14 6 26 4 124 4 4 69 3\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:masked_lm_positions: 18 20 21 22 29 43 57 75 76 81 88 89 95 96 111 120 122 124 125 0\n",
            "INFO:tensorflow:masked_lm_ids: 549 5 97 9 184 41 6328 8 247 14 6 112 243 172 36 6 39 989 15 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] oh god his skull ain t even fused together man [SEP] gettin scoop ##ed up [MASK] the field nothingness [SEP]\n",
            "INFO:tensorflow:input_ids: 2 56 137 87 3871 382 12 142 10858 269 89 3 1228 10734 161 51 4 7 1313 20323 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 1 16 19 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 56 119 19260 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] ryan [MASK] t swear [SEP] [MASK] [MASK] true is that he s exhausted so we [MASK] to try to find [MASK] [MASK] other than the haldol to help him [MASK] good thank you i ll have dr presto ##p ##nik write up a prescription hello chris sister [SEP]\n",
            "INFO:tensorflow:input_ids: 2 7219 4 12 1264 3 4 4 381 17 14 21 8 5696 43 20 4 9 247 9 164 4 4 183 197 7 14686 9 153 60 4 71 147 5 6 47 29 473 8255 2402 16861 918 51 10 5495 212 3214 417 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 2 6 7 16 21 22 30 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 30 16 17 128 10 2883 416 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 0\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] evacuation warning evacuation warning evacuation [MASK] [MASK] warning [SEP] the area get everyone away [MASK] the [MASK] core [MASK] radiation gear is on [MASK] 10 commander javio wait sir this prowler is non operational there s [MASK] corpses component in the in the fro ##oni ##um drive causing instabilit ##ies in the stabilise ##r it may have been fixed if i could just have a look very [MASK] chiana i ##misunderstanding the commander was [MASK] be the last one to evacuate it s funny i believe just the opposite we could both leave continue our earlier discussion it won t work twice what was that the tissue sample [MASK] succeeded then i m [MASK] going [MASK] die [MASK] you once said to me [MASK] [MASK] die [SEP]\n",
            "INFO:tensorflow:input_ids: 2 4928 1862 4928 1862 4928 4 4 1862 3 7 1383 44 405 163 4 7 4 5494 4 6821 4315 17 24 4 554 2178 9531 150 205 22 2987 17 5086 11124 50 8 4 8381 10938 19 7 19 7 10266 22658 3816 617 6253 19079 1364 19 7 14707 440 11 245 29 109 2281 59 6 93 46 29 10 82 124 4 1662 6 4968 7 2178 35 4 37 7 200 61 9 4374 11 8 519 6 223 46 7 4632 20 93 378 218 1337 114 1582 2966 11 158 12 154 1066 16 35 14 7 3798 3737 4 7544 110 6 26 4 106 4 307 4 5 348 135 9 18 4 4 307 3\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:masked_lm_positions: 6 7 15 17 19 24 37 38 41 61 68 71 75 109 114 116 118 124 125 0\n",
            "INFO:tensorflow:masked_lm_ids: 1862 4928 81 6995 1413 1459 10 15200 7 6 71 171 9 941 39 9 83 5 79 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 0\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] i [MASK] seen them come and go [MASK] day [MASK] laid eyes on you i said [MASK] s gonna be a good one you can t be good [MASK] you love it [SEP] do you [MASK] me governor do you [SEP]\n",
            "INFO:tensorflow:input_ids: 2 6 4 314 105 58 13 52 4 160 4 2162 413 24 5 6 135 4 8 80 37 10 71 61 5 34 12 37 71 4 5 113 11 3 31 5 4 18 3541 31 5 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 2 8 10 17 29 36 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 73 7 6 53 836 447 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] but [MASK] decides she doesn t want [MASK] [MASK] either until she find ##s out that ernie does [MASK] then she s got somethin to bargain with ernie had this bright red full ##y restored [UNK] camar ##o i mean [MASK] beauty and she wants it so he [MASK] his car for her daughter novalee oh my god [MASK] he told me that [MASK] i knew he was the pick of the litter i [MASK] so happy [MASK] finally found me a [MASK] oh i m also [MASK] what both [SEP] eight [MASK] nine then eight then nine eight and then nine eight and nine but the one i ve been used to [MASK] [MASK] eight and a half then eight then eight and a half [SEP]\n",
            "INFO:tensorflow:input_ids: 2 42 4 6084 53 208 12 72 4 4 625 412 53 164 57 55 14 3471 210 4 110 53 8 63 1111 9 9573 41 3471 97 22 1837 751 481 221 8169 1 19183 299 6 132 4 863 13 53 401 11 43 21 4 87 201 28 68 515 1322 56 25 137 4 21 196 18 14 4 6 313 21 35 7 505 15 7 3979 6 4 43 294 4 715 312 18 10 4 56 6 26 465 4 16 378 3 728 4 928 110 728 110 928 728 13 110 928 728 13 928 42 7 61 6 73 109 352 9 4 4 728 13 10 450 110 728 110 728 13 10 450 3\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:masked_lm_positions: 2 8 9 19 28 35 36 41 49 59 64 75 78 83 88 93 102 114 115 0\n",
            "INFO:tensorflow:masked_lm_ids: 53 3360 2589 13 3471 8169 1 10 13414 90 1322 139 6 1886 1828 13 928 17 184 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] a [MASK] emph ##ys ##ema it [MASK] nothin serious you know he makes that noise whenever he [MASK] ##s himself who s thorny forney forney hull brilliant man lives in the libary brilliant there [MASK] no tellin what he might have done in this [MASK] if he d been allowed to finish his [MASK] why didn t he the lord gives us obstacles i hope you re not hungry cause it s tuesday friday [MASK] the best breakfast day they [MASK] [MASK] i m lexie lexie coop hi hi where d all these flowers come from all [MASK] you re a big celebrity they re reporters here [MASK] up see tv they ve [MASK] here all night [SEP] i know [MASK] thing [SEP]\n",
            "INFO:tensorflow:input_ids: 2 10 4 28519 11253 18918 11 4 1203 581 5 36 21 472 14 1840 2144 21 4 57 693 75 8 14946 1349 1349 6502 2352 89 652 19 7 12168 2352 50 4 23 4348 16 21 272 29 277 19 22 4 59 21 86 109 1702 9 1001 87 4 78 112 12 21 7 630 1497 99 13696 6 375 5 33 39 948 338 11 8 2355 1274 4 7 253 1506 160 49 4 4 6 26 3487 3487 13448 217 217 103 86 40 151 1318 58 81 40 4 5 33 10 148 8109 49 33 7734 45 4 51 77 593 49 73 4 45 40 169 3 6 36 4 143 3 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 2 7 18 35 45 54 59 60 75 81 82 87 98 100 102 108 114 119 121 0\n",
            "INFO:tensorflow:masked_lm_ids: 111 8 11982 8 242 20406 7 630 17 29 5120 13448 120 33 148 3634 109 6 10 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] is this because i forgot how can you forget something like that how can you forget something like what don t you remember who died don t you remember who died don t [MASK] [MASK] [MASK] after where are we where [MASK] [MASK] meditate to be why is this happening to me [MASK] you were starting to become cold i think your time has come now time to open the box don t listen to her she s a [MASK] do not [MASK] a word he says listen to [MASK] i [MASK] [MASK] it s difficult to grasp what s going on and you are what the other side the other side [SEP] i am [MASK] other side [MASK] other side of [MASK] but not you [SEP]\n",
            "INFO:tensorflow:input_ids: 2 17 22 134 6 852 67 34 5 410 116 48 14 67 34 5 410 116 48 16 30 12 5 239 75 575 30 12 5 239 75 575 30 12 4 4 4 202 103 38 20 103 4 4 11092 9 37 78 17 22 846 9 18 4 5 101 1162 9 619 620 6 74 27 92 126 58 65 92 9 326 7 1069 30 12 229 9 68 53 8 10 4 31 39 4 10 458 21 369 229 9 4 6 4 4 11 8 1599 9 7797 16 8 106 24 13 5 38 16 7 183 555 7 183 555 3 6 139 4 183 555 4 183 555 15 4 42 39 5 3\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:masked_lm_positions: 25 31 34 35 36 37 42 43 44 53 68 80 83 90 92 93 116 119 123 0\n",
            "INFO:tensorflow:masked_lm_ids: 575 575 5 239 75 575 5 108 72 134 9 1656 223 18 234 14 7 7 68 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 0\n",
            "INFO:tensorflow:*** Writing to output files ***\n",
            "INFO:tensorflow:  pretraining_data/shard_0003.tfrecord\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] what good s having [MASK] unless you spend it huh this is this is so extravagant i i m embarrassed don midnight take up smoking but if it s good enough for ##toilet duke of windsor [MASK] well i m really [MASK] and my little bernard shaw is so [MASK] no [SEP] you to [MASK] how unbelievabl ##y proud i am of you david this is [MASK] insignificant maybe i m talking out of turn here but [MASK] ve just developed [MASK] much and i i just feel that refugee belong in a [MASK] of society and [MASK] culture now [MASK] s frustrating because you should continue growing you magog be branch ##ing out there [MASK] paris and rome and the great [MASK] houses and museums [SEP]\n",
            "INFO:tensorflow:input_ids: 2 16 71 8 400 4 836 5 908 11 227 22 17 22 17 43 13014 6 6 26 2713 30 2925 94 51 1572 42 59 11 8 71 235 28 25052 3674 15 5794 4 69 6 26 108 4 13 25 111 6019 6851 17 43 4 23 3 5 9 4 67 6773 221 989 6 139 15 5 627 22 17 4 7066 155 6 26 271 55 15 289 45 42 4 73 46 3801 4 140 13 6 6 46 199 14 6896 1675 19 10 4 15 2732 13 4 2937 65 4 8 10652 134 5 133 1337 1496 5 7216 37 6459 123 55 50 4 1542 13 2629 13 7 167 4 2992 13 6490 3\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:masked_lm_positions: 5 16 22 33 37 42 50 55 60 67 78 82 90 94 98 101 110 116 123 0\n",
            "INFO:tensorflow:masked_lm_ids: 1389 13014 12 7 54 7353 7066 36 6 39 5 43 5 242 434 11 133 8 2290 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 0\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] you wanna be a socialite [SEP] [MASK] [MASK] nuts [SEP]\n",
            "INFO:tensorflow:input_ids: 2 5 243 37 10 9581 3 4 4 1168 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 7 8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 38 5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] i [MASK] t know [SEP] i guess sometimes these ##claimed sort [MASK] come [MASK] in their own kook ##y sort of way [SEP]\n",
            "INFO:tensorflow:input_ids: 2 6 4 12 36 3 6 297 566 151 21001 669 4 58 4 19 193 248 10837 221 669 15 118 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 2 10 12 14 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 30 185 15 269 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] [MASK] distance running well you just run [MASK] [MASK] [MASK] back home then [MASK] you don t understand can t you just show me how mongolian get out of a head lock [SEP] i knew it damn [SEP]\n",
            "INFO:tensorflow:input_ids: 2 4 3732 662 69 5 46 384 4 4 4 85 174 110 4 5 30 12 234 34 12 5 46 257 18 67 14817 44 55 15 10 267 1258 3 6 313 11 290 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 1 8 9 10 14 26 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 177 40 7 118 23 9 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] did [MASK] know that in some cultures if your children shame you you re allowed [MASK] have them executed [MASK] i feel shamed oh harold i m [MASK] swinging an ax [SEP] i just [MASK] [MASK] to play pit with me i mean is that so horrible honey [MASK] s like that old saying if you love something set it [MASK] maybe [MASK] we try algeria to force it [MASK] ll tightrope to us well all right there you go [MASK] [MASK] free let em free let em go laughs [MASK] get back here all right bill you pick wait why does [MASK] get to pick because i don t trust [MASK] just pick fast don t try to feel around for your paper i m [SEP]\n",
            "INFO:tensorflow:input_ids: 2 76 4 36 14 19 102 6501 59 27 645 1431 5 5 33 1702 4 29 105 4998 4 6 199 15997 56 4675 6 26 4 7844 98 8796 3 6 46 4 4 9 391 2916 41 18 6 132 17 14 43 1983 380 4 8 48 14 186 341 59 5 113 116 556 11 4 155 4 20 247 18178 9 868 11 4 47 18405 9 99 69 40 54 50 5 52 4 4 606 70 428 606 70 428 52 1432 4 44 85 45 40 54 629 5 505 150 78 210 4 44 9 505 134 6 30 12 493 4 46 505 580 30 12 247 9 199 190 28 27 822 6 26 3\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:masked_lm_positions: 2 16 20 28 35 36 49 57 61 63 66 70 72 76 81 82 91 103 112 0\n",
            "INFO:tensorflow:masked_lm_ids: 5 9 69 39 72 105 11 113 606 59 39 49 58 40 49 33 77 21 5 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 0\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] i mean once people caught on that that was possible you had madison avenue [MASK] into the job at ##t hire ##s wai to [MASK] people that now its you know its become a very mainstream message [MASK] had a hunch that it was [MASK] to change things in a way that [MASK] advent of [MASK] [MASK] ##bi ##quit ##aka of the automobile changed things it changed how [MASK] dressed how we eat it affect ##s these [MASK] affect everything i mean this was a [MASK] [MASK] technologies it was just a matter of spraying [MASK] the hairs ##pr ##ay and slapping on some lip gloss and this thing was gonna walk you [MASK] they were gonna [MASK] cute [SEP] your what [SEP]\n",
            "INFO:tensorflow:input_ids: 2 6 132 348 138 915 24 14 14 35 901 5 97 6342 4328 4 173 7 309 64 180 2727 57 11564 9 4 138 14 65 773 5 36 773 619 10 124 23564 1164 4 97 10 4379 14 11 35 4 9 406 185 19 10 118 14 4 26278 15 4 4 11992 14518 20654 15 7 8455 725 185 11 725 67 4 1520 67 20 358 11 3474 57 151 4 3474 189 6 132 22 35 10 4 4 3777 11 35 46 10 325 15 24515 4 7 15084 25762 4109 13 13835 24 102 4897 8327 13 22 143 35 80 491 5 4 49 101 80 4 1147 3 27 16 3 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 1 15 23 25 38 45 53 55 56 57 60 69 78 86 87 96 114 116 118 0\n",
            "INFO:tensorflow:masked_lm_ids: 6 295 138 104 6 106 7 15 7 957 221 20 185 11753 1676 24 36 101 37 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] here goes nothing the maru has activated the fms her sensors [MASK] emitt ##ing at full [MASK] harper did a great job [MASK] i topless t know [MASK] i would swear she [MASK] me apparently [MASK] re not the only one [MASK] m [MASK] a sensor contact at extreme range [MASK] closing fast on the maru [SEP] tag you re it keep it together beka [MASK] t let him spook you multipl ##e [MASK] four missiles headed for the maru all [MASK] he s taken the bait all ahead full ready missile barr ##age on my mark collision [MASK] collision alert tell [MASK] [MASK] i don t know [MASK] s not going to make it covering fire now come on come on [MASK] just one more [SEP]\n",
            "INFO:tensorflow:input_ids: 2 45 460 166 7 4892 126 5467 7 22021 68 4689 4 25133 123 64 481 4 5249 76 10 167 309 4 6 11462 12 36 4 6 107 1264 53 4 18 1568 4 33 39 7 130 61 4 26 4 10 7762 1286 64 8605 3245 4 3491 580 24 7 4892 3 3010 5 33 11 179 11 269 9216 4 12 70 60 12714 5 7863 178 4 393 7108 1403 28 7 4892 40 4 21 8 741 7 2274 40 513 481 345 5094 15921 3259 24 25 1256 15426 4 15426 2248 104 4 4 6 30 12 36 4 8 39 106 9 122 11 5585 558 65 58 24 58 24 4 46 61 129 3\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:masked_lm_positions: 12 17 23 25 28 33 36 42 44 51 53 66 74 82 99 103 104 109 123 0\n",
            "INFO:tensorflow:masked_lm_ids: 38 547 59 112 182 101 5 6 292 13 580 30 11222 54 2248 18 116 53 3275 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 0\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] must [MASK] [MASK] [MASK] with your head i never killed anyone before what [MASK] re a schizo who s wanted [MASK] murder and you re laying in bed next to me [MASK] ##g like [MASK] [UNK] year old girl on the phone you freaked [MASK] out man but [MASK] [MASK] your family [SEP] then you could have a hell of a problem mr byrnes [SEP]\n",
            "INFO:tensorflow:input_ids: 2 191 4 4 4 41 27 267 6 117 356 403 195 16 4 33 10 3907 75 8 246 4 768 13 5 33 4225 19 635 288 9 18 4 1271 48 4 1 422 186 231 24 7 443 5 4329 4 55 89 42 4 4 27 258 3 110 5 93 29 10 220 15 10 291 209 3175 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 2 3 4 14 21 32 35 45 49 50 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 37 14484 1271 5 28 12075 10 18 16 62 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] so [MASK] is that so terrible but when we talked about making it we [MASK] gonna hit [MASK] big move to [MASK] swim eat [MASK] crabs we can [MASK] a place in palm beach palm beach is ritzy [SEP] [MASK] at the dog track every day i just wanna [MASK] as far [MASK] from frenchy fox the topless wonder as i can be hey you were beautiful as [MASK] fox [MASK] d come out [MASK] you ##vee give it mine little of this those days are over and our accountants want us to be twice as big next [MASK] what good is that if i can t [MASK] a cheeseburger ray [MASK] [MASK] charming tonight i ve seen you when you wanna turn on the charm [SEP]\n",
            "INFO:tensorflow:input_ids: 2 43 4 17 14 43 803 42 90 20 1079 62 520 11 20 4 80 446 4 148 295 9 4 1345 358 4 8512 20 34 4 10 224 19 3120 1195 3120 1195 17 13106 3 4 64 7 304 1434 261 160 6 46 243 4 83 470 4 81 755 2087 7 11462 757 83 6 34 37 95 5 101 365 83 4 2087 4 86 58 55 4 5 16081 131 11 367 111 15 22 181 360 38 120 13 114 4470 72 99 9 37 1066 83 148 288 4 16 71 17 14 59 6 34 12 4 10 9744 658 4 4 3114 420 6 73 314 5 90 5 243 289 24 7 3042 3\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:masked_lm_positions: 2 15 18 22 25 29 40 50 53 69 71 75 77 80 95 99 108 112 113 0\n",
            "INFO:tensorflow:masked_lm_ids: 16 101 11 2104 2038 44 37 37 163 755 5 239 86 10 1066 422 44 141 37 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 0\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] [MASK] [MASK] thing get that [MASK] [SEP] the math motherfucker damn jeez fellas i think i found it what false alarm oh damn ##als [MASK] got a star on my car and one on my chest a gun [MASK] my hip and the right to arrest i [MASK] a guy who s the boss on this highway so [MASK] out what you re doin when you re drivin my way if you break the law [MASK] ll [MASK] from [MASK] i know i m [MASK] for the state i m the highway patrol well [MASK] ll know me when you see me cos my door ##ugie painted white my siren a screamin and my [MASK] ##ing red light i work all day and i [MASK] all [SEP]\n",
            "INFO:tensorflow:input_ids: 2 4 4 143 44 14 4 3 7 4150 935 290 2901 1533 6 74 6 312 11 16 2436 1730 56 290 20565 4 63 10 945 24 25 201 13 61 24 25 1775 10 461 4 25 3875 13 7 54 9 1420 6 4 10 203 75 8 7 782 24 22 2267 43 4 55 16 5 33 530 90 5 33 4716 25 118 59 5 448 7 569 4 47 4 81 4 6 36 6 26 4 28 7 588 6 26 7 2267 1830 69 4 47 36 18 90 5 77 18 1138 25 396 18540 1652 642 25 3646 10 8345 13 25 4 123 751 474 6 154 40 160 13 6 4 40 3\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:masked_lm_positions: 1 2 6 24 25 33 39 42 48 59 76 77 78 80 85 95 106 115 125 0\n",
            "INFO:tensorflow:masked_lm_ids: 44 14 143 2217 6 61 24 13 26 324 5 47 240 18 2474 5 8 3183 154 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] the child in you will never [MASK] the power [MASK] believe is yours alone no one can ever take that child away see the world for a while [MASK] the eyes ofa child you will see so much wonder in your days all that [MASK] believed when you were small [MASK] still the dreams that guide us all the mirror shows you ve [MASK] but reflect ##s the child in you lfthe ##re s one thing i could say toyou [SEP] [MASK] make [MASK] dream come true you can do almost anything if youjust believe if youjust believe if only only only [MASK] believe bonja [MASK] is an [MASK] [MASK] everyone [MASK] dream very often [MASK] do [MASK] a lot but mine has already been withered [SEP]\n",
            "INFO:tensorflow:input_ids: 2 7 498 19 5 79 117 4 7 547 4 223 17 506 368 23 61 34 225 94 14 498 163 77 7 242 28 10 327 4 7 413 19273 498 5 79 77 43 140 757 19 27 360 40 14 4 2360 90 5 101 744 4 159 7 1249 14 2572 99 40 7 1845 1937 5 73 4 42 6882 57 7 498 19 5 31580 667 8 61 143 6 93 96 23169 3 4 122 4 628 58 381 5 34 31 583 162 59 12471 223 59 12471 223 59 130 130 130 4 223 23584 4 17 98 4 4 405 4 628 124 1425 4 31 4 10 232 42 367 126 323 109 17628 3\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:masked_lm_positions: 7 10 11 15 29 35 45 51 64 82 84 91 103 106 109 110 112 116 118 0\n",
            "INFO:tensorflow:masked_lm_ids: 3614 9 223 23 262 79 5 38 1447 9 1802 583 5 628 4101 143 79 6 628 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 0\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] all right all right lt [MASK] me he s he s here ln the red snappy ##y nova uh that s [MASK] sir [MASK] over all right as soon as he leaves you call me with the owners addresses spirits [MASK] i ll [MASK] those forms [MASK] over sir damn boss always on my [MASK] i [MASK] three here mm hmm thankyou huh randall raines donny astricky otto halli ##well should i set up some tails uh uh they re too smart for that they ll just dump the cars no we [MASK] get them [MASK] the mercedes [SEP] i won t give up now sorry i [MASK] clear some things up they gotta [MASK] cleared [MASK] what the fuck so [MASK] stiff straight in line [SEP]\n",
            "INFO:tensorflow:input_ids: 2 40 54 40 54 543 4 18 21 8 21 8 45 3617 7 751 15638 221 7901 144 14 8 4 205 4 120 40 54 83 433 83 21 1739 5 165 18 41 7 4303 2698 4075 4 6 47 4 181 2829 4 120 205 290 782 184 24 25 4 6 4 222 45 476 503 3338 227 892 1023 1234 6565 1042 10222 4120 133 6 556 51 102 4194 144 144 49 33 121 719 28 14 49 47 46 2058 7 641 23 20 4 44 105 4 7 1860 3 6 158 12 131 51 65 136 6 4 587 102 185 51 49 198 4 2856 4 16 7 275 43 4 3678 674 19 517 3\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:masked_lm_positions: 6 16 22 24 40 41 42 44 47 55 57 75 93 96 102 108 115 117 122 0\n",
            "INFO:tensorflow:masked_lm_ids: 8 17073 5314 626 40 54 6 44 54 459 63 102 47 41 12 198 44 51 283 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] [MASK] i think about it and i often do it s like film images clearly etched and [MASK] lines i suffer from something called retrospect ##ive jealousy during [MASK] intensely erotic interl ##ude in paris i questioned marianne joking ##ly [MASK] her previous lovers she was trusting and walked right [MASK] the trap she was touched [MASK] my interest [MASK] told [MASK] in detail about her relationship with [MASK] how [MASK] some circumstances she achieve ##d an intensity off [MASK] that she d never experienced either before or since that cut deeply [MASK] [MASK] a small but infected wound and that disastrous night the wound broke [MASK] and there was nothing i could do i see [MASK] her face [SEP] i m not [MASK] with you [SEP]\n",
            "INFO:tensorflow:input_ids: 2 4 6 74 62 11 13 6 1425 31 11 8 48 1122 6510 2287 8508 13 4 2102 6 1834 81 116 343 14956 5146 6844 1211 4 14211 6108 28987 25388 19 1542 6 5040 2123 2249 353 4 68 3537 2858 53 35 12874 13 1308 54 4 7 3442 53 35 2433 4 25 2129 4 196 4 19 4660 62 68 1068 41 4 67 4 102 3148 53 4067 206 98 10068 125 4 14 53 86 117 4542 625 195 100 374 14 372 3355 4 4 10 744 42 5969 1773 13 14 10966 169 7 1773 956 4 13 50 35 166 6 93 31 6 77 4 68 332 3 6 26 39 4 41 5 3\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:masked_lm_positions: 1 18 27 29 41 42 51 57 60 62 69 71 76 80 93 94 107 117 124 0\n",
            "INFO:tensorflow:masked_lm_ids: 90 7 6844 114 62 68 173 119 13 18 1215 423 206 25853 13 1207 326 2123 262 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] guess who s back in [MASK] circle of trust pull the string and i [MASK] wink [MASK] [MASK] i m your puppet look at jack whoa oh poor kevin looks lonely maybe i should [MASK] him to dance what [MASK] you think [MASK] [SEP] cancelled [SEP]\n",
            "INFO:tensorflow:input_ids: 2 297 75 8 85 19 4 1799 15 493 626 7 5328 13 6 4 5079 4 4 6 26 27 8236 82 64 418 379 56 538 1105 308 1678 155 6 133 4 60 9 659 16 4 5 74 4 3 7802 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 6 15 17 18 35 40 43 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 7 47 64 5 279 31 149 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] sighing [MASK] to me okay it [MASK] over dickie it [MASK] over [SEP] why don t i drive [SEP]\n",
            "INFO:tensorflow:input_ids: 2 5257 4 9 18 91 11 4 120 1348 11 4 120 3 78 30 12 6 617 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 2 7 11 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 229 8 8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] to me terribly sorry very sorry overhearing get posts [MASK] customer account listing and you [MASK] me an evaluation report [SEP] cover the other [MASK] i want charlie arrested on [MASK] hey irene you just keep walking now [SEP]\n",
            "INFO:tensorflow:input_ids: 2 9 18 4040 136 124 136 17471 44 21015 4 2892 2529 24125 13 5 4 18 98 11379 805 3 947 7 183 4 6 72 276 2241 24 4 95 639 5 46 179 1133 65 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 7 9 10 16 25 31 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 205 18 7 44 362 1865 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] then [MASK] and i the action occurs let s say in a circus ah no i m not dating i member i m in love with another character the magician who is much older than i ##zie [MASK] [MASK] the [MASK] did i tell you he plays the violin all i want is to see her to hold her to forgive her what happened with rafael nothing too great my best friend [MASK] the winner is why can [MASK] everything [MASK] back to what [MASK] used to be i [MASK] [MASK] everything tomorrow i ll file for [MASK] let s go home let go [MASK] leave [MASK] [MASK] easy [SEP] leave me alone drop that i m going to nag myself lucia i [MASK] another man [SEP]\n",
            "INFO:tensorflow:input_ids: 2 110 4 13 6 7 1242 12759 70 8 96 19 10 4063 477 23 6 26 39 2220 6 2875 6 26 19 113 41 255 2408 7 5781 75 17 140 1902 197 6 11173 4 4 7 4 76 6 104 5 21 1835 7 4393 40 6 72 17 9 77 68 9 252 68 9 746 68 16 260 41 10984 166 121 167 25 253 264 4 7 1886 17 78 34 4 189 4 85 9 16 4 352 9 37 6 4 4 189 407 6 47 1611 28 4 70 8 52 174 70 52 4 218 4 4 320 3 218 18 368 724 14 6 26 106 9 4755 377 14671 6 4 255 89 3\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:masked_lm_positions: 2 21 37 38 39 41 60 73 79 81 85 90 91 98 105 107 108 120 124 0\n",
            "INFO:tensorflow:masked_lm_ids: 5 74 139 13 1835 4393 9 13 12 52 11 196 60 2492 752 18 368 228 570 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 0\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] cracker ##jack what are you lookin at i happen to be noticing [MASK] i was lookin at you a [MASK] of times [MASK] [MASK] for the first time i had the thought that in [MASK] very very strange way you [MASK] a sweet face it s off beat in a kind of bizarre you know i don t know how to explain misbehaving [SEP] you mean it s a mainzer may i m trying [MASK] say a nice thing cos [MASK] being a relative of [MASK] i never [MASK] classified [MASK] as [MASK] human type female so i i i was married a long time [MASK] right it was a really tragic story [MASK] my husband [MASK] was dyslexic [MASK] the only thing he could [MASK] [SEP]\n",
            "INFO:tensorflow:input_ids: 2 5362 7293 16 38 5 1039 64 6 436 9 37 8675 4 6 35 1039 64 5 10 4 15 514 4 4 28 7 175 92 6 97 7 171 14 19 4 124 124 683 118 5 4 10 644 332 11 8 125 743 19 10 251 15 3906 5 36 6 30 12 36 67 9 774 21539 3 5 132 11 8 10 21421 245 6 26 329 4 96 10 194 143 1138 4 293 10 4314 15 4 6 117 4 9557 4 83 4 605 1646 2356 43 6 6 6 35 462 10 177 92 4 54 11 35 10 108 3606 542 4 25 632 4 35 13012 4 7 130 143 21 93 4 3\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:masked_lm_positions: 13 20 23 24 35 41 63 70 75 81 86 89 91 93 106 114 117 120 126 0\n",
            "INFO:tensorflow:masked_lm_ids: 5 604 420 1138 10 63 11 6409 9 5 755 195 5 10 349 134 1042 13 2511 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 0\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] woman s voice [MASK] four lev axle [MASK] you ni ##v dee can [UNK] groaning [UNK] darvo se ##d four red do 10 screeching fastened ray i m hungry frenchy learn something there s nothing to learn this is [MASK] honor james lived [MASK] the ##pyjamas leader stupid married to betty grab ##le yes [SEP] to the to the to the [MASK] teat [SEP]\n",
            "INFO:tensorflow:input_ids: 2 270 8 701 4 393 11922 14835 4 5 5280 3576 2507 34 1 1903 1 14109 2833 206 393 751 31 554 3094 18545 658 6 26 948 755 673 116 50 8 166 9 673 22 17 4 912 2320 950 4 7 29665 1055 419 462 9 2523 890 2233 84 3 9 7 9 7 9 7 4 9951 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_positions: 4 8 25 38 40 41 44 46 60 62 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_ids: 1 11605 4326 22 103 1567 75 1578 9 9951 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:tokens: [CLS] am just a dog [SEP] well don [MASK] forget to [MASK] i guess we could always appear atyour trial [MASK] animat ##ed character witnesses no rocky bullwinkle it s your [MASK] to get [MASK] new york by [UNK] 00 pm tomorrow rocky karen we can t leave ##you ##now hey i m [MASK] [MASK] agent remember i ll straighten this out and catch up with you [MASK] but we ve never been [MASK] the real world before oh you ll do great i know it hey what about our faces oh sorry wait you can t just [MASK] them on the side ofthe [MASK] [MASK] that they don t even know [MASK] they are don t [MASK] sup am they re animals they ll for ##age [SEP]\n",
            "INFO:tensorflow:input_ids: 2 139 46 10 304 3 69 30 4 410 9 4 6 297 20 93 184 3062 15107 1887 4 25603 161 2408 5006 23 1267 1093 11 8 27 4 9 44 4 170 789 119 1 614 6059 407 1267 1102 20 34 12 218 527 4143 95 6 26 4 4 634 239 6 47 3638 22 55 13 548 51 41 5 4 42 20 73 117 109 4 7 256 242 195 56 5 47 31 167 6 36 11 95 16 62 114 3357 56 136 150 5 34 12 46 4 105 24 7 555 2847 4 4 14 49 30 12 142 36 4 49 38 30 12 4 20317 139 49 33 2112 49 47 28 3259 3\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
            "INFO:tensorflow:masked_lm_positions: 8 11 12 20 31 34 53 54 62 64 67 73 98 104 105 108 112 117 118 0\n",
            "INFO:tensorflow:masked_lm_ids: 12 918 6 83 1857 9 98 1181 13 51 386 19 218 1071 48 30 103 351 526 0\n",
            "INFO:tensorflow:masked_lm_weights: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0\n",
            "INFO:tensorflow:next_sentence_labels: 1\n",
            "INFO:tensorflow:Wrote 94522 total instances\n",
            "INFO:tensorflow:Wrote 88775 total instances\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZKnW_hVxPoP6",
        "colab_type": "text"
      },
      "source": [
        "## Step 6: setting up persistent storage\n",
        "\n",
        "To preserve our hard-earned assets, we will persist them to Google Cloud Storage. Provided that you have created the GCS bucket, this should be simple.\n",
        "\n",
        "We will create two directories in GCS, one for the data and one for the model.\n",
        "In the model directory, we will put the model vocabulary and configuration file.\n",
        "\n",
        "**Configure your BUCKET_NAME variable here before proceeding, otherwise the model and data will not be saved.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gDtrt68QQIHs",
        "colab_type": "code",
        "cellView": "both",
        "colab": {}
      },
      "source": [
        "BUCKET_NAME = \"bert_resourses\" #@param {type:\"string\"}\n",
        "MODEL_DIR = \"bert_model\" #@param {type:\"string\"}\n",
        "tf.gfile.MkDir(MODEL_DIR)\n",
        "\n",
        "if not BUCKET_NAME:\n",
        "  log.warning(\"WARNING: BUCKET_NAME is not set. \"\n",
        "              \"You will not be able to train the model.\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YigCcV-hSHVH",
        "colab_type": "text"
      },
      "source": [
        "Below is the sample hyperparameter configuration for BERT-base. Change at your own risk."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SEpSGpUKReKW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# use this for BERT-base\n",
        "\n",
        "bert_base_config = {\n",
        "  \"attention_probs_dropout_prob\": 0.1, \n",
        "  \"directionality\": \"bidi\", \n",
        "  \"hidden_act\": \"gelu\", \n",
        "  \"hidden_dropout_prob\": 0.1, \n",
        "  \"hidden_size\": 768, \n",
        "  \"initializer_range\": 0.02, \n",
        "  \"intermediate_size\": 3072, \n",
        "  \"max_position_embeddings\": 512, \n",
        "  \"num_attention_heads\": 12, \n",
        "  \"num_hidden_layers\": 12, \n",
        "  \"pooler_fc_size\": 768, \n",
        "  \"pooler_num_attention_heads\": 12, \n",
        "  \"pooler_num_fc_layers\": 3, \n",
        "  \"pooler_size_per_head\": 128, \n",
        "  \"pooler_type\": \"first_token_transform\", \n",
        "  \"type_vocab_size\": 2, \n",
        "  \"vocab_size\": VOC_SIZE\n",
        "}\n",
        "\n",
        "with open(\"{}/bert_config.json\".format(MODEL_DIR), \"w\") as fo:\n",
        "  json.dump(bert_base_config, fo, indent=2)\n",
        "  \n",
        "with open(\"{}/{}\".format(MODEL_DIR, VOC_FNAME), \"w\") as fo:\n",
        "  for token in bert_vocab:\n",
        "    fo.write(token+\"\\n\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "txgrEDugRG48",
        "colab_type": "code",
        "outputId": "da579e4f-ed50-476e-a928-7b544b0e805c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 153
        }
      },
      "source": [
        "if BUCKET_NAME:\n",
        "  !gsutil -m cp -r $MODEL_DIR $PRETRAINING_DIR gs://$BUCKET_NAME"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Copying file://pretraining_data/shard_0000.tfrecord [Content-Type=application/octet-stream]...\n",
            "Copying file://bert_model/vocab.txt [Content-Type=text/plain]...\n",
            "Copying file://bert_model/bert_config.json [Content-Type=application/json]...\n",
            "Copying file://pretraining_data/shard_0003.tfrecord [Content-Type=application/octet-stream]...\n",
            "Copying file://pretraining_data/shard_0001.tfrecord [Content-Type=application/octet-stream]...\n",
            "Copying file://pretraining_data/shard_0002.tfrecord [Content-Type=application/octet-stream]...\n",
            "/ [6/6 files][269.4 MiB/269.4 MiB] 100% Done                                    \n",
            "Operation completed over 6 objects/269.4 MiB.                                    \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3DL6xuCAYPrp",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3gdQEOzhYmSh",
        "colab_type": "text"
      },
      "source": [
        "## Step 7: training the model\n",
        "\n",
        "We are almost ready to begin training our model. If you wish  to continue an interrupted training run, you may skip steps 2-6 and proceed from here.\n",
        "\n",
        "**Make sure that you have set the BUCKET_NAME here as well.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yXAuzsJfYrio",
        "colab_type": "code",
        "cellView": "both",
        "outputId": "486c27a7-bd28-4197-d42c-b7ab84e0ca6e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "BUCKET_NAME = \"bert_resourses\" #@param {type:\"string\"}\n",
        "MODEL_DIR = \"bert_model\" #@param {type:\"string\"}\n",
        "PRETRAINING_DIR = \"pretraining_data\" #@param {type:\"string\"}\n",
        "VOC_FNAME = \"vocab.txt\" #@param {type:\"string\"}\n",
        "\n",
        "# Input data pipeline config\n",
        "TRAIN_BATCH_SIZE = 128 #@param {type:\"integer\"}\n",
        "MAX_PREDICTIONS = 20 #@param {type:\"integer\"}\n",
        "MAX_SEQ_LENGTH = 128 #@param {type:\"integer\"}\n",
        "MASKED_LM_PROB = 0.15 #@param\n",
        "\n",
        "# Training procedure config\n",
        "EVAL_BATCH_SIZE = 64\n",
        "LEARNING_RATE = 2e-5\n",
        "TRAIN_STEPS = 1000000 #@param {type:\"integer\"}\n",
        "SAVE_CHECKPOINTS_STEPS = 2500 #@param {type:\"integer\"}\n",
        "NUM_TPU_CORES = 8\n",
        "\n",
        "if BUCKET_NAME:\n",
        "  BUCKET_PATH = \"gs://{}\".format(BUCKET_NAME)\n",
        "else:\n",
        "  BUCKET_PATH = \".\"\n",
        "\n",
        "BERT_GCS_DIR = \"{}/{}\".format(BUCKET_PATH, MODEL_DIR)\n",
        "DATA_GCS_DIR = \"{}/{}\".format(BUCKET_PATH, PRETRAINING_DIR)\n",
        "\n",
        "VOCAB_FILE = os.path.join(BERT_GCS_DIR, VOC_FNAME)\n",
        "CONFIG_FILE = os.path.join(BERT_GCS_DIR, \"bert_config.json\")\n",
        "\n",
        "INIT_CHECKPOINT = tf.train.latest_checkpoint(BERT_GCS_DIR)\n",
        "\n",
        "bert_config = modeling.BertConfig.from_json_file(CONFIG_FILE)\n",
        "input_files = tf.gfile.Glob(os.path.join(DATA_GCS_DIR,'*tfrecord'))\n",
        "\n",
        "log.info(\"Using checkpoint: {}\".format(INIT_CHECKPOINT))\n",
        "log.info(\"Using {} data shards\".format(len(input_files)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2019-05-09 14:14:46,902 :  Using checkpoint: None\n",
            "2019-05-09 14:14:46,903 :  Using 4 data shards\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zwwF-WqcZHUG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MQTFjITdd53F",
        "colab_type": "text"
      },
      "source": [
        "Prepare the training run configuration, build the estimator and input function, power up the bass cannon."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CMahsqUnZ55z",
        "colab_type": "code",
        "outputId": "b7f6300f-7725-492a-977b-c499a12c4ea8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 258
        }
      },
      "source": [
        "model_fn = model_fn_builder(\n",
        "      bert_config=bert_config,\n",
        "      init_checkpoint=INIT_CHECKPOINT,\n",
        "      learning_rate=LEARNING_RATE,\n",
        "      num_train_steps=TRAIN_STEPS,\n",
        "      num_warmup_steps=10,\n",
        "      use_tpu=USE_TPU,\n",
        "      use_one_hot_embeddings=True)\n",
        "\n",
        "tpu_cluster_resolver = tf.contrib.cluster_resolver.TPUClusterResolver(TPU_ADDRESS)\n",
        "\n",
        "run_config = tf.contrib.tpu.RunConfig(\n",
        "    cluster=tpu_cluster_resolver,\n",
        "    model_dir=BERT_GCS_DIR,\n",
        "    save_checkpoints_steps=SAVE_CHECKPOINTS_STEPS,\n",
        "    tpu_config=tf.contrib.tpu.TPUConfig(\n",
        "        iterations_per_loop=SAVE_CHECKPOINTS_STEPS,\n",
        "        num_shards=NUM_TPU_CORES,\n",
        "        per_host_input_for_training=tf.contrib.tpu.InputPipelineConfig.PER_HOST_V2))\n",
        "\n",
        "estimator = tf.contrib.tpu.TPUEstimator(\n",
        "    use_tpu=USE_TPU,\n",
        "    model_fn=model_fn,\n",
        "    config=run_config,\n",
        "    train_batch_size=TRAIN_BATCH_SIZE,\n",
        "    eval_batch_size=EVAL_BATCH_SIZE)\n",
        "  \n",
        "train_input_fn = input_fn_builder(\n",
        "        input_files=input_files,\n",
        "        max_seq_length=MAX_SEQ_LENGTH,\n",
        "        max_predictions_per_seq=MAX_PREDICTIONS,\n",
        "        is_training=True)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2019-05-09 14:14:54,593 :  Estimator's model_fn (<function model_fn_builder.<locals>.model_fn at 0x7effebc4f950>) includes params argument, but params are not passed to Estimator.\n",
            "2019-05-09 14:14:54,604 :  Using config: {'_model_dir': 'gs://bert_resourses/bert_model', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 2500, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "cluster_def {\n",
            "  job {\n",
            "    name: \"worker\"\n",
            "    tasks {\n",
            "      key: 0\n",
            "      value: \"10.60.104.26:8470\"\n",
            "    }\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7effebd46e10>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': 'grpc://10.60.104.26:8470', '_evaluation_master': 'grpc://10.60.104.26:8470', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=2500, num_shards=8, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None), '_cluster': <tensorflow.python.distribute.cluster_resolver.tpu_cluster_resolver.TPUClusterResolver object at 0x7effec2a52b0>}\n",
            "2019-05-09 14:14:54,609 :  _TPUContext: eval_on_tpu True\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kNt5ykopeIYB",
        "colab_type": "text"
      },
      "source": [
        "Fire!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JrCuEbr6dv8U",
        "colab_type": "code",
        "outputId": "d18248f8-c22d-44b1-9369-75448f5e9b64",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 8843
        }
      },
      "source": [
        "estimator.train(input_fn=train_input_fn, max_steps=TRAIN_STEPS)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2019-05-09 14:15:07,866 :  Querying Tensorflow master (grpc://10.60.104.26:8470) for TPU system metadata.\n",
            "2019-05-09 14:15:07,893 :  Found TPU system:\n",
            "2019-05-09 14:15:07,895 :  *** Num TPU Cores: 8\n",
            "2019-05-09 14:15:07,896 :  *** Num TPU Workers: 1\n",
            "2019-05-09 14:15:07,901 :  *** Num TPU Cores Per Worker: 8\n",
            "2019-05-09 14:15:07,903 :  *** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:CPU:0, CPU, -1, 14897432867444107573)\n",
            "2019-05-09 14:15:07,905 :  *** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 17179869184, 276397939913914612)\n",
            "2019-05-09 14:15:07,907 :  *** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:0, TPU, 17179869184, 13057281094532086746)\n",
            "2019-05-09 14:15:07,909 :  *** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:1, TPU, 17179869184, 14081917870675593807)\n",
            "2019-05-09 14:15:07,911 :  *** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:2, TPU, 17179869184, 2105416731698971008)\n",
            "2019-05-09 14:15:07,912 :  *** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:3, TPU, 17179869184, 13479851194553820680)\n",
            "2019-05-09 14:15:07,913 :  *** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:4, TPU, 17179869184, 12266856648231000659)\n",
            "2019-05-09 14:15:07,914 :  *** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:5, TPU, 17179869184, 13865385848777702521)\n",
            "2019-05-09 14:15:07,917 :  *** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:6, TPU, 17179869184, 17873047070900026345)\n",
            "2019-05-09 14:15:07,919 :  *** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:7, TPU, 17179869184, 1381190866836211913)\n",
            "2019-05-09 14:15:07,919 :  *** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU_SYSTEM:0, TPU_SYSTEM, 17179869184, 11134852893159600164)\n",
            "2019-05-09 14:15:07,942 :  From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py:435: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Colocations handled automatically by placer.\n",
            "2019-05-09 14:15:07,977 :  Calling model_fn.\n",
            "2019-05-09 14:15:07,993 :  From /content/bert/run_pretraining.py:368: parallel_interleave (from tensorflow.contrib.data.python.ops.interleave_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use `tf.data.experimental.parallel_interleave(...)`.\n",
            "2019-05-09 14:15:08,042 :  From /content/bert/run_pretraining.py:385: map_and_batch (from tensorflow.contrib.data.python.ops.batching) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use `tf.data.experimental.map_and_batch(...)`.\n",
            "2019-05-09 14:15:08,065 :  From /content/bert/run_pretraining.py:400: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.cast instead.\n",
            "2019-05-09 14:15:08,162 :  *** Features ***\n",
            "2019-05-09 14:15:08,164 :    name = input_ids, shape = (16, 128)\n",
            "2019-05-09 14:15:08,164 :    name = input_mask, shape = (16, 128)\n",
            "2019-05-09 14:15:08,171 :    name = masked_lm_ids, shape = (16, 20)\n",
            "2019-05-09 14:15:08,176 :    name = masked_lm_positions, shape = (16, 20)\n",
            "2019-05-09 14:15:08,178 :    name = masked_lm_weights, shape = (16, 20)\n",
            "2019-05-09 14:15:08,180 :    name = next_sentence_labels, shape = (16, 1)\n",
            "2019-05-09 14:15:08,181 :    name = segment_ids, shape = (16, 128)\n",
            "2019-05-09 14:15:08,304 :  From bert/modeling.py:358: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n",
            "2019-05-09 14:15:08,332 :  From bert/modeling.py:671: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use keras.layers.dense instead.\n",
            "2019-05-09 14:15:12,297 :  **** Trainable Variables ****\n",
            "2019-05-09 14:15:12,298 :    name = bert/embeddings/word_embeddings:0, shape = (32000, 768)\n",
            "2019-05-09 14:15:12,299 :    name = bert/embeddings/token_type_embeddings:0, shape = (2, 768)\n",
            "2019-05-09 14:15:12,306 :    name = bert/embeddings/position_embeddings:0, shape = (512, 768)\n",
            "2019-05-09 14:15:12,307 :    name = bert/embeddings/LayerNorm/beta:0, shape = (768,)\n",
            "2019-05-09 14:15:12,309 :    name = bert/embeddings/LayerNorm/gamma:0, shape = (768,)\n",
            "2019-05-09 14:15:12,310 :    name = bert/encoder/layer_0/attention/self/query/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,312 :    name = bert/encoder/layer_0/attention/self/query/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,314 :    name = bert/encoder/layer_0/attention/self/key/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,316 :    name = bert/encoder/layer_0/attention/self/key/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,319 :    name = bert/encoder/layer_0/attention/self/value/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,320 :    name = bert/encoder/layer_0/attention/self/value/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,321 :    name = bert/encoder/layer_0/attention/output/dense/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,324 :    name = bert/encoder/layer_0/attention/output/dense/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,326 :    name = bert/encoder/layer_0/attention/output/LayerNorm/beta:0, shape = (768,)\n",
            "2019-05-09 14:15:12,328 :    name = bert/encoder/layer_0/attention/output/LayerNorm/gamma:0, shape = (768,)\n",
            "2019-05-09 14:15:12,329 :    name = bert/encoder/layer_0/intermediate/dense/kernel:0, shape = (768, 3072)\n",
            "2019-05-09 14:15:12,331 :    name = bert/encoder/layer_0/intermediate/dense/bias:0, shape = (3072,)\n",
            "2019-05-09 14:15:12,333 :    name = bert/encoder/layer_0/output/dense/kernel:0, shape = (3072, 768)\n",
            "2019-05-09 14:15:12,334 :    name = bert/encoder/layer_0/output/dense/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,337 :    name = bert/encoder/layer_0/output/LayerNorm/beta:0, shape = (768,)\n",
            "2019-05-09 14:15:12,339 :    name = bert/encoder/layer_0/output/LayerNorm/gamma:0, shape = (768,)\n",
            "2019-05-09 14:15:12,340 :    name = bert/encoder/layer_1/attention/self/query/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,342 :    name = bert/encoder/layer_1/attention/self/query/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,343 :    name = bert/encoder/layer_1/attention/self/key/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,345 :    name = bert/encoder/layer_1/attention/self/key/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,346 :    name = bert/encoder/layer_1/attention/self/value/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,348 :    name = bert/encoder/layer_1/attention/self/value/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,349 :    name = bert/encoder/layer_1/attention/output/dense/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,351 :    name = bert/encoder/layer_1/attention/output/dense/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,352 :    name = bert/encoder/layer_1/attention/output/LayerNorm/beta:0, shape = (768,)\n",
            "2019-05-09 14:15:12,354 :    name = bert/encoder/layer_1/attention/output/LayerNorm/gamma:0, shape = (768,)\n",
            "2019-05-09 14:15:12,355 :    name = bert/encoder/layer_1/intermediate/dense/kernel:0, shape = (768, 3072)\n",
            "2019-05-09 14:15:12,357 :    name = bert/encoder/layer_1/intermediate/dense/bias:0, shape = (3072,)\n",
            "2019-05-09 14:15:12,358 :    name = bert/encoder/layer_1/output/dense/kernel:0, shape = (3072, 768)\n",
            "2019-05-09 14:15:12,360 :    name = bert/encoder/layer_1/output/dense/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,361 :    name = bert/encoder/layer_1/output/LayerNorm/beta:0, shape = (768,)\n",
            "2019-05-09 14:15:12,363 :    name = bert/encoder/layer_1/output/LayerNorm/gamma:0, shape = (768,)\n",
            "2019-05-09 14:15:12,364 :    name = bert/encoder/layer_2/attention/self/query/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,366 :    name = bert/encoder/layer_2/attention/self/query/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,367 :    name = bert/encoder/layer_2/attention/self/key/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,373 :    name = bert/encoder/layer_2/attention/self/key/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,374 :    name = bert/encoder/layer_2/attention/self/value/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,376 :    name = bert/encoder/layer_2/attention/self/value/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,377 :    name = bert/encoder/layer_2/attention/output/dense/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,378 :    name = bert/encoder/layer_2/attention/output/dense/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,381 :    name = bert/encoder/layer_2/attention/output/LayerNorm/beta:0, shape = (768,)\n",
            "2019-05-09 14:15:12,383 :    name = bert/encoder/layer_2/attention/output/LayerNorm/gamma:0, shape = (768,)\n",
            "2019-05-09 14:15:12,384 :    name = bert/encoder/layer_2/intermediate/dense/kernel:0, shape = (768, 3072)\n",
            "2019-05-09 14:15:12,386 :    name = bert/encoder/layer_2/intermediate/dense/bias:0, shape = (3072,)\n",
            "2019-05-09 14:15:12,388 :    name = bert/encoder/layer_2/output/dense/kernel:0, shape = (3072, 768)\n",
            "2019-05-09 14:15:12,390 :    name = bert/encoder/layer_2/output/dense/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,391 :    name = bert/encoder/layer_2/output/LayerNorm/beta:0, shape = (768,)\n",
            "2019-05-09 14:15:12,393 :    name = bert/encoder/layer_2/output/LayerNorm/gamma:0, shape = (768,)\n",
            "2019-05-09 14:15:12,394 :    name = bert/encoder/layer_3/attention/self/query/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,396 :    name = bert/encoder/layer_3/attention/self/query/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,397 :    name = bert/encoder/layer_3/attention/self/key/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,399 :    name = bert/encoder/layer_3/attention/self/key/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,401 :    name = bert/encoder/layer_3/attention/self/value/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,402 :    name = bert/encoder/layer_3/attention/self/value/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,404 :    name = bert/encoder/layer_3/attention/output/dense/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,405 :    name = bert/encoder/layer_3/attention/output/dense/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,408 :    name = bert/encoder/layer_3/attention/output/LayerNorm/beta:0, shape = (768,)\n",
            "2019-05-09 14:15:12,409 :    name = bert/encoder/layer_3/attention/output/LayerNorm/gamma:0, shape = (768,)\n",
            "2019-05-09 14:15:12,411 :    name = bert/encoder/layer_3/intermediate/dense/kernel:0, shape = (768, 3072)\n",
            "2019-05-09 14:15:12,412 :    name = bert/encoder/layer_3/intermediate/dense/bias:0, shape = (3072,)\n",
            "2019-05-09 14:15:12,414 :    name = bert/encoder/layer_3/output/dense/kernel:0, shape = (3072, 768)\n",
            "2019-05-09 14:15:12,416 :    name = bert/encoder/layer_3/output/dense/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,417 :    name = bert/encoder/layer_3/output/LayerNorm/beta:0, shape = (768,)\n",
            "2019-05-09 14:15:12,420 :    name = bert/encoder/layer_3/output/LayerNorm/gamma:0, shape = (768,)\n",
            "2019-05-09 14:15:12,421 :    name = bert/encoder/layer_4/attention/self/query/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,422 :    name = bert/encoder/layer_4/attention/self/query/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,423 :    name = bert/encoder/layer_4/attention/self/key/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,425 :    name = bert/encoder/layer_4/attention/self/key/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,428 :    name = bert/encoder/layer_4/attention/self/value/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,429 :    name = bert/encoder/layer_4/attention/self/value/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,431 :    name = bert/encoder/layer_4/attention/output/dense/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,432 :    name = bert/encoder/layer_4/attention/output/dense/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,434 :    name = bert/encoder/layer_4/attention/output/LayerNorm/beta:0, shape = (768,)\n",
            "2019-05-09 14:15:12,436 :    name = bert/encoder/layer_4/attention/output/LayerNorm/gamma:0, shape = (768,)\n",
            "2019-05-09 14:15:12,437 :    name = bert/encoder/layer_4/intermediate/dense/kernel:0, shape = (768, 3072)\n",
            "2019-05-09 14:15:12,439 :    name = bert/encoder/layer_4/intermediate/dense/bias:0, shape = (3072,)\n",
            "2019-05-09 14:15:12,441 :    name = bert/encoder/layer_4/output/dense/kernel:0, shape = (3072, 768)\n",
            "2019-05-09 14:15:12,443 :    name = bert/encoder/layer_4/output/dense/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,444 :    name = bert/encoder/layer_4/output/LayerNorm/beta:0, shape = (768,)\n",
            "2019-05-09 14:15:12,446 :    name = bert/encoder/layer_4/output/LayerNorm/gamma:0, shape = (768,)\n",
            "2019-05-09 14:15:12,448 :    name = bert/encoder/layer_5/attention/self/query/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,450 :    name = bert/encoder/layer_5/attention/self/query/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,451 :    name = bert/encoder/layer_5/attention/self/key/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,453 :    name = bert/encoder/layer_5/attention/self/key/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,455 :    name = bert/encoder/layer_5/attention/self/value/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,457 :    name = bert/encoder/layer_5/attention/self/value/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,458 :    name = bert/encoder/layer_5/attention/output/dense/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,460 :    name = bert/encoder/layer_5/attention/output/dense/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,461 :    name = bert/encoder/layer_5/attention/output/LayerNorm/beta:0, shape = (768,)\n",
            "2019-05-09 14:15:12,463 :    name = bert/encoder/layer_5/attention/output/LayerNorm/gamma:0, shape = (768,)\n",
            "2019-05-09 14:15:12,464 :    name = bert/encoder/layer_5/intermediate/dense/kernel:0, shape = (768, 3072)\n",
            "2019-05-09 14:15:12,466 :    name = bert/encoder/layer_5/intermediate/dense/bias:0, shape = (3072,)\n",
            "2019-05-09 14:15:12,467 :    name = bert/encoder/layer_5/output/dense/kernel:0, shape = (3072, 768)\n",
            "2019-05-09 14:15:12,469 :    name = bert/encoder/layer_5/output/dense/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,470 :    name = bert/encoder/layer_5/output/LayerNorm/beta:0, shape = (768,)\n",
            "2019-05-09 14:15:12,471 :    name = bert/encoder/layer_5/output/LayerNorm/gamma:0, shape = (768,)\n",
            "2019-05-09 14:15:12,474 :    name = bert/encoder/layer_6/attention/self/query/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,475 :    name = bert/encoder/layer_6/attention/self/query/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,477 :    name = bert/encoder/layer_6/attention/self/key/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,478 :    name = bert/encoder/layer_6/attention/self/key/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,479 :    name = bert/encoder/layer_6/attention/self/value/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,480 :    name = bert/encoder/layer_6/attention/self/value/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,482 :    name = bert/encoder/layer_6/attention/output/dense/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,483 :    name = bert/encoder/layer_6/attention/output/dense/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,485 :    name = bert/encoder/layer_6/attention/output/LayerNorm/beta:0, shape = (768,)\n",
            "2019-05-09 14:15:12,486 :    name = bert/encoder/layer_6/attention/output/LayerNorm/gamma:0, shape = (768,)\n",
            "2019-05-09 14:15:12,487 :    name = bert/encoder/layer_6/intermediate/dense/kernel:0, shape = (768, 3072)\n",
            "2019-05-09 14:15:12,489 :    name = bert/encoder/layer_6/intermediate/dense/bias:0, shape = (3072,)\n",
            "2019-05-09 14:15:12,491 :    name = bert/encoder/layer_6/output/dense/kernel:0, shape = (3072, 768)\n",
            "2019-05-09 14:15:12,492 :    name = bert/encoder/layer_6/output/dense/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,494 :    name = bert/encoder/layer_6/output/LayerNorm/beta:0, shape = (768,)\n",
            "2019-05-09 14:15:12,495 :    name = bert/encoder/layer_6/output/LayerNorm/gamma:0, shape = (768,)\n",
            "2019-05-09 14:15:12,497 :    name = bert/encoder/layer_7/attention/self/query/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,498 :    name = bert/encoder/layer_7/attention/self/query/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,501 :    name = bert/encoder/layer_7/attention/self/key/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,502 :    name = bert/encoder/layer_7/attention/self/key/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,503 :    name = bert/encoder/layer_7/attention/self/value/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,505 :    name = bert/encoder/layer_7/attention/self/value/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,507 :    name = bert/encoder/layer_7/attention/output/dense/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,509 :    name = bert/encoder/layer_7/attention/output/dense/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,510 :    name = bert/encoder/layer_7/attention/output/LayerNorm/beta:0, shape = (768,)\n",
            "2019-05-09 14:15:12,512 :    name = bert/encoder/layer_7/attention/output/LayerNorm/gamma:0, shape = (768,)\n",
            "2019-05-09 14:15:12,513 :    name = bert/encoder/layer_7/intermediate/dense/kernel:0, shape = (768, 3072)\n",
            "2019-05-09 14:15:12,515 :    name = bert/encoder/layer_7/intermediate/dense/bias:0, shape = (3072,)\n",
            "2019-05-09 14:15:12,516 :    name = bert/encoder/layer_7/output/dense/kernel:0, shape = (3072, 768)\n",
            "2019-05-09 14:15:12,518 :    name = bert/encoder/layer_7/output/dense/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,519 :    name = bert/encoder/layer_7/output/LayerNorm/beta:0, shape = (768,)\n",
            "2019-05-09 14:15:12,521 :    name = bert/encoder/layer_7/output/LayerNorm/gamma:0, shape = (768,)\n",
            "2019-05-09 14:15:12,522 :    name = bert/encoder/layer_8/attention/self/query/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,523 :    name = bert/encoder/layer_8/attention/self/query/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,524 :    name = bert/encoder/layer_8/attention/self/key/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,527 :    name = bert/encoder/layer_8/attention/self/key/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,528 :    name = bert/encoder/layer_8/attention/self/value/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,530 :    name = bert/encoder/layer_8/attention/self/value/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,532 :    name = bert/encoder/layer_8/attention/output/dense/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,533 :    name = bert/encoder/layer_8/attention/output/dense/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,534 :    name = bert/encoder/layer_8/attention/output/LayerNorm/beta:0, shape = (768,)\n",
            "2019-05-09 14:15:12,535 :    name = bert/encoder/layer_8/attention/output/LayerNorm/gamma:0, shape = (768,)\n",
            "2019-05-09 14:15:12,536 :    name = bert/encoder/layer_8/intermediate/dense/kernel:0, shape = (768, 3072)\n",
            "2019-05-09 14:15:12,540 :    name = bert/encoder/layer_8/intermediate/dense/bias:0, shape = (3072,)\n",
            "2019-05-09 14:15:12,542 :    name = bert/encoder/layer_8/output/dense/kernel:0, shape = (3072, 768)\n",
            "2019-05-09 14:15:12,543 :    name = bert/encoder/layer_8/output/dense/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,544 :    name = bert/encoder/layer_8/output/LayerNorm/beta:0, shape = (768,)\n",
            "2019-05-09 14:15:12,545 :    name = bert/encoder/layer_8/output/LayerNorm/gamma:0, shape = (768,)\n",
            "2019-05-09 14:15:12,546 :    name = bert/encoder/layer_9/attention/self/query/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,547 :    name = bert/encoder/layer_9/attention/self/query/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,549 :    name = bert/encoder/layer_9/attention/self/key/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,550 :    name = bert/encoder/layer_9/attention/self/key/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,552 :    name = bert/encoder/layer_9/attention/self/value/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,553 :    name = bert/encoder/layer_9/attention/self/value/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,554 :    name = bert/encoder/layer_9/attention/output/dense/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,555 :    name = bert/encoder/layer_9/attention/output/dense/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,556 :    name = bert/encoder/layer_9/attention/output/LayerNorm/beta:0, shape = (768,)\n",
            "2019-05-09 14:15:12,558 :    name = bert/encoder/layer_9/attention/output/LayerNorm/gamma:0, shape = (768,)\n",
            "2019-05-09 14:15:12,559 :    name = bert/encoder/layer_9/intermediate/dense/kernel:0, shape = (768, 3072)\n",
            "2019-05-09 14:15:12,561 :    name = bert/encoder/layer_9/intermediate/dense/bias:0, shape = (3072,)\n",
            "2019-05-09 14:15:12,562 :    name = bert/encoder/layer_9/output/dense/kernel:0, shape = (3072, 768)\n",
            "2019-05-09 14:15:12,564 :    name = bert/encoder/layer_9/output/dense/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,565 :    name = bert/encoder/layer_9/output/LayerNorm/beta:0, shape = (768,)\n",
            "2019-05-09 14:15:12,566 :    name = bert/encoder/layer_9/output/LayerNorm/gamma:0, shape = (768,)\n",
            "2019-05-09 14:15:12,567 :    name = bert/encoder/layer_10/attention/self/query/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,569 :    name = bert/encoder/layer_10/attention/self/query/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,570 :    name = bert/encoder/layer_10/attention/self/key/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,571 :    name = bert/encoder/layer_10/attention/self/key/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,572 :    name = bert/encoder/layer_10/attention/self/value/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,573 :    name = bert/encoder/layer_10/attention/self/value/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,575 :    name = bert/encoder/layer_10/attention/output/dense/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,576 :    name = bert/encoder/layer_10/attention/output/dense/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,578 :    name = bert/encoder/layer_10/attention/output/LayerNorm/beta:0, shape = (768,)\n",
            "2019-05-09 14:15:12,579 :    name = bert/encoder/layer_10/attention/output/LayerNorm/gamma:0, shape = (768,)\n",
            "2019-05-09 14:15:12,581 :    name = bert/encoder/layer_10/intermediate/dense/kernel:0, shape = (768, 3072)\n",
            "2019-05-09 14:15:12,582 :    name = bert/encoder/layer_10/intermediate/dense/bias:0, shape = (3072,)\n",
            "2019-05-09 14:15:12,584 :    name = bert/encoder/layer_10/output/dense/kernel:0, shape = (3072, 768)\n",
            "2019-05-09 14:15:12,585 :    name = bert/encoder/layer_10/output/dense/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,586 :    name = bert/encoder/layer_10/output/LayerNorm/beta:0, shape = (768,)\n",
            "2019-05-09 14:15:12,588 :    name = bert/encoder/layer_10/output/LayerNorm/gamma:0, shape = (768,)\n",
            "2019-05-09 14:15:12,589 :    name = bert/encoder/layer_11/attention/self/query/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,591 :    name = bert/encoder/layer_11/attention/self/query/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,593 :    name = bert/encoder/layer_11/attention/self/key/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,594 :    name = bert/encoder/layer_11/attention/self/key/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,596 :    name = bert/encoder/layer_11/attention/self/value/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,597 :    name = bert/encoder/layer_11/attention/self/value/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,598 :    name = bert/encoder/layer_11/attention/output/dense/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,599 :    name = bert/encoder/layer_11/attention/output/dense/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,600 :    name = bert/encoder/layer_11/attention/output/LayerNorm/beta:0, shape = (768,)\n",
            "2019-05-09 14:15:12,602 :    name = bert/encoder/layer_11/attention/output/LayerNorm/gamma:0, shape = (768,)\n",
            "2019-05-09 14:15:12,603 :    name = bert/encoder/layer_11/intermediate/dense/kernel:0, shape = (768, 3072)\n",
            "2019-05-09 14:15:12,605 :    name = bert/encoder/layer_11/intermediate/dense/bias:0, shape = (3072,)\n",
            "2019-05-09 14:15:12,606 :    name = bert/encoder/layer_11/output/dense/kernel:0, shape = (3072, 768)\n",
            "2019-05-09 14:15:12,608 :    name = bert/encoder/layer_11/output/dense/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,609 :    name = bert/encoder/layer_11/output/LayerNorm/beta:0, shape = (768,)\n",
            "2019-05-09 14:15:12,610 :    name = bert/encoder/layer_11/output/LayerNorm/gamma:0, shape = (768,)\n",
            "2019-05-09 14:15:12,611 :    name = bert/pooler/dense/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,612 :    name = bert/pooler/dense/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,613 :    name = cls/predictions/transform/dense/kernel:0, shape = (768, 768)\n",
            "2019-05-09 14:15:12,614 :    name = cls/predictions/transform/dense/bias:0, shape = (768,)\n",
            "2019-05-09 14:15:12,615 :    name = cls/predictions/transform/LayerNorm/beta:0, shape = (768,)\n",
            "2019-05-09 14:15:12,617 :    name = cls/predictions/transform/LayerNorm/gamma:0, shape = (768,)\n",
            "2019-05-09 14:15:12,619 :    name = cls/predictions/output_bias:0, shape = (32000,)\n",
            "2019-05-09 14:15:12,620 :    name = cls/seq_relationship/output_weights:0, shape = (2, 768)\n",
            "2019-05-09 14:15:12,621 :    name = cls/seq_relationship/output_bias:0, shape = (2,)\n",
            "2019-05-09 14:15:12,639 :  From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/learning_rate_decay_v2.py:321: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Deprecated in favor of operator or tf.math.divide.\n",
            "2019-05-09 14:15:26,955 :  Create CheckpointSaverHook.\n",
            "2019-05-09 14:15:27,372 :  Done calling model_fn.\n",
            "2019-05-09 14:15:32,799 :  TPU job name worker\n",
            "2019-05-09 14:15:34,459 :  Graph was finalized.\n",
            "2019-05-09 14:15:41,844 :  Running local_init_op.\n",
            "2019-05-09 14:15:42,481 :  Done running local_init_op.\n",
            "2019-05-09 14:15:55,485 :  Saving checkpoints for 0 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 14:16:23,703 :  Initialized dataset iterators in 0 seconds\n",
            "2019-05-09 14:16:23,705 :  Installing graceful shutdown hook.\n",
            "2019-05-09 14:16:23,718 :  Creating heartbeat manager for ['/job:worker/replica:0/task:0/device:CPU:0']\n",
            "2019-05-09 14:16:23,738 :  Configuring worker heartbeat: shutdown_mode: WAIT_FOR_COORDINATOR\n",
            "\n",
            "2019-05-09 14:16:23,751 :  Init TPU system\n",
            "2019-05-09 14:16:31,110 :  Initialized TPU in 7 seconds\n",
            "2019-05-09 14:16:31,112 :  Starting infeed thread controller.\n",
            "2019-05-09 14:16:31,112 :  Starting outfeed thread controller.\n",
            "2019-05-09 14:16:31,762 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 14:16:31,764 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 14:22:32,677 :  Saving checkpoints for 2500 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 14:23:04,050 :  loss = 6.0122037, step = 2500\n",
            "2019-05-09 14:23:04,054 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 14:23:04,055 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 14:28:22,058 :  Saving checkpoints for 5000 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 14:28:50,566 :  loss = 6.2573056, step = 5000 (346.516 sec)\n",
            "2019-05-09 14:28:50,569 :  global_step/sec: 7.21468\n",
            "2019-05-09 14:28:50,575 :  examples/sec: 923.479\n",
            "2019-05-09 14:28:50,581 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 14:28:50,582 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 14:34:02,961 :  Saving checkpoints for 7500 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 14:34:32,186 :  loss = 5.985909, step = 7500 (341.620 sec)\n",
            "2019-05-09 14:34:32,189 :  global_step/sec: 7.31805\n",
            "2019-05-09 14:34:32,192 :  examples/sec: 936.71\n",
            "2019-05-09 14:34:32,197 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 14:34:32,198 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 14:39:44,577 :  Saving checkpoints for 10000 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 14:40:12,658 :  loss = 6.368071, step = 10000 (340.472 sec)\n",
            "2019-05-09 14:40:12,660 :  global_step/sec: 7.34277\n",
            "2019-05-09 14:40:12,662 :  examples/sec: 939.874\n",
            "2019-05-09 14:40:12,665 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 14:40:12,666 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 14:45:25,103 :  Saving checkpoints for 12500 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 14:45:48,683 :  From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:966: remove_checkpoint (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use standard file APIs to delete files with this prefix.\n",
            "2019-05-09 14:45:55,106 :  loss = 5.108742, step = 12500 (342.448 sec)\n",
            "2019-05-09 14:45:55,109 :  global_step/sec: 7.30037\n",
            "2019-05-09 14:45:55,113 :  examples/sec: 934.448\n",
            "2019-05-09 14:45:55,116 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 14:45:55,118 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 14:51:07,400 :  Saving checkpoints for 15000 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 14:51:37,887 :  loss = 4.6821427, step = 15000 (342.781 sec)\n",
            "2019-05-09 14:51:37,890 :  global_step/sec: 7.29329\n",
            "2019-05-09 14:51:37,897 :  examples/sec: 933.541\n",
            "2019-05-09 14:51:37,900 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 14:51:37,901 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 14:56:50,395 :  Saving checkpoints for 17500 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 14:57:22,161 :  loss = 3.934165, step = 17500 (344.274 sec)\n",
            "2019-05-09 14:57:22,164 :  global_step/sec: 7.26166\n",
            "2019-05-09 14:57:22,165 :  examples/sec: 929.492\n",
            "2019-05-09 14:57:22,167 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 14:57:22,169 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 15:02:34,523 :  Saving checkpoints for 20000 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 15:03:05,309 :  loss = 4.0289288, step = 20000 (343.148 sec)\n",
            "2019-05-09 15:03:05,312 :  global_step/sec: 7.28547\n",
            "2019-05-09 15:03:05,313 :  examples/sec: 932.541\n",
            "2019-05-09 15:03:05,316 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 15:03:05,317 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 15:08:17,726 :  Saving checkpoints for 22500 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 15:08:48,959 :  loss = 3.656248, step = 22500 (343.649 sec)\n",
            "2019-05-09 15:08:48,961 :  global_step/sec: 7.27486\n",
            "2019-05-09 15:08:48,964 :  examples/sec: 931.183\n",
            "2019-05-09 15:08:48,971 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 15:08:48,972 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 15:14:01,316 :  Saving checkpoints for 25000 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 15:14:35,780 :  loss = 3.6549108, step = 25000 (346.821 sec)\n",
            "2019-05-09 15:14:35,783 :  global_step/sec: 7.20832\n",
            "2019-05-09 15:14:35,784 :  examples/sec: 922.665\n",
            "2019-05-09 15:14:35,790 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 15:14:35,791 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 15:19:48,093 :  Saving checkpoints for 27500 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 15:20:19,353 :  loss = 3.5114233, step = 27500 (343.573 sec)\n",
            "2019-05-09 15:20:19,355 :  global_step/sec: 7.27649\n",
            "2019-05-09 15:20:19,356 :  examples/sec: 931.391\n",
            "2019-05-09 15:20:19,359 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 15:20:19,361 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 15:25:31,839 :  Saving checkpoints for 30000 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 15:26:08,795 :  loss = 3.5775118, step = 30000 (349.442 sec)\n",
            "2019-05-09 15:26:08,800 :  global_step/sec: 7.15421\n",
            "2019-05-09 15:26:08,802 :  examples/sec: 915.738\n",
            "2019-05-09 15:26:08,804 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 15:26:08,805 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 15:31:21,263 :  Saving checkpoints for 32500 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 15:31:56,244 :  loss = 3.12237, step = 32500 (347.450 sec)\n",
            "2019-05-09 15:31:56,247 :  global_step/sec: 7.19533\n",
            "2019-05-09 15:31:56,249 :  examples/sec: 921.002\n",
            "2019-05-09 15:31:56,252 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 15:31:56,254 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 15:37:08,642 :  Saving checkpoints for 35000 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 15:37:40,297 :  loss = 3.3346393, step = 35000 (344.052 sec)\n",
            "2019-05-09 15:37:40,299 :  global_step/sec: 7.26634\n",
            "2019-05-09 15:37:40,301 :  examples/sec: 930.091\n",
            "2019-05-09 15:37:40,304 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 15:37:40,306 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 15:42:52,640 :  Saving checkpoints for 37500 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 15:43:21,764 :  loss = 3.0102618, step = 37500 (341.468 sec)\n",
            "2019-05-09 15:43:21,767 :  global_step/sec: 7.32134\n",
            "2019-05-09 15:43:21,770 :  examples/sec: 937.131\n",
            "2019-05-09 15:43:21,776 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 15:43:21,778 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 15:48:34,224 :  Saving checkpoints for 40000 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 15:49:08,178 :  loss = 2.5819876, step = 40000 (346.414 sec)\n",
            "2019-05-09 15:49:08,181 :  global_step/sec: 7.21681\n",
            "2019-05-09 15:49:08,182 :  examples/sec: 923.751\n",
            "2019-05-09 15:49:08,184 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 15:49:08,187 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 15:54:20,595 :  Saving checkpoints for 42500 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 15:54:53,060 :  loss = 2.5919535, step = 42500 (344.882 sec)\n",
            "2019-05-09 15:54:53,064 :  global_step/sec: 7.24884\n",
            "2019-05-09 15:54:53,067 :  examples/sec: 927.851\n",
            "2019-05-09 15:54:53,071 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 15:54:53,072 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 16:00:05,531 :  Saving checkpoints for 45000 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 16:00:41,861 :  loss = 2.3893905, step = 45000 (348.801 sec)\n",
            "2019-05-09 16:00:41,863 :  global_step/sec: 7.16744\n",
            "2019-05-09 16:00:41,865 :  examples/sec: 917.432\n",
            "2019-05-09 16:00:41,868 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 16:00:41,870 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 16:05:54,269 :  Saving checkpoints for 47500 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 16:06:28,084 :  loss = 2.461726, step = 47500 (346.224 sec)\n",
            "2019-05-09 16:06:28,087 :  global_step/sec: 7.22077\n",
            "2019-05-09 16:06:28,088 :  examples/sec: 924.259\n",
            "2019-05-09 16:06:28,090 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 16:06:28,093 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 16:11:40,486 :  Saving checkpoints for 50000 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 16:12:10,243 :  loss = 2.246158, step = 50000 (342.158 sec)\n",
            "2019-05-09 16:12:10,245 :  global_step/sec: 7.30655\n",
            "2019-05-09 16:12:10,247 :  examples/sec: 935.238\n",
            "2019-05-09 16:12:10,250 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 16:12:10,252 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 16:17:22,578 :  Saving checkpoints for 52500 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 16:17:53,758 :  loss = 2.6802454, step = 52500 (343.515 sec)\n",
            "2019-05-09 16:17:53,761 :  global_step/sec: 7.2777\n",
            "2019-05-09 16:17:53,762 :  examples/sec: 931.546\n",
            "2019-05-09 16:17:53,765 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 16:17:53,767 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 16:23:06,219 :  Saving checkpoints for 55000 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 16:23:39,312 :  loss = 1.8007338, step = 55000 (345.554 sec)\n",
            "2019-05-09 16:23:39,315 :  global_step/sec: 7.23475\n",
            "2019-05-09 16:23:39,316 :  examples/sec: 926.048\n",
            "2019-05-09 16:23:39,319 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 16:23:39,320 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 16:28:51,777 :  Saving checkpoints for 57500 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 16:29:23,387 :  loss = 1.7874498, step = 57500 (344.075 sec)\n",
            "2019-05-09 16:29:23,392 :  global_step/sec: 7.26581\n",
            "2019-05-09 16:29:23,394 :  examples/sec: 930.024\n",
            "2019-05-09 16:29:23,396 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 16:29:23,398 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 16:34:35,702 :  Saving checkpoints for 60000 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 16:35:06,218 :  loss = 2.891969, step = 60000 (342.831 sec)\n",
            "2019-05-09 16:35:06,221 :  global_step/sec: 7.29227\n",
            "2019-05-09 16:35:06,222 :  examples/sec: 933.411\n",
            "2019-05-09 16:35:06,228 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 16:35:06,231 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 16:40:18,628 :  Saving checkpoints for 62500 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 16:40:48,239 :  loss = 1.8042313, step = 62500 (342.021 sec)\n",
            "2019-05-09 16:40:48,242 :  global_step/sec: 7.30949\n",
            "2019-05-09 16:40:48,243 :  examples/sec: 935.615\n",
            "2019-05-09 16:40:48,247 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 16:40:48,248 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 16:46:00,681 :  Saving checkpoints for 65000 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 16:46:30,404 :  loss = 1.8897341, step = 65000 (342.164 sec)\n",
            "2019-05-09 16:46:30,409 :  global_step/sec: 7.30638\n",
            "2019-05-09 16:46:30,413 :  examples/sec: 935.216\n",
            "2019-05-09 16:46:30,418 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 16:46:30,419 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 16:51:42,851 :  Saving checkpoints for 67500 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 16:52:12,737 :  loss = 1.5435848, step = 67500 (342.334 sec)\n",
            "2019-05-09 16:52:12,740 :  global_step/sec: 7.30287\n",
            "2019-05-09 16:52:12,745 :  examples/sec: 934.767\n",
            "2019-05-09 16:52:12,748 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 16:52:12,750 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 16:57:25,161 :  Saving checkpoints for 70000 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 16:57:53,416 :  loss = 1.5164316, step = 70000 (340.679 sec)\n",
            "2019-05-09 16:57:53,419 :  global_step/sec: 7.33829\n",
            "2019-05-09 16:57:53,421 :  examples/sec: 939.301\n",
            "2019-05-09 16:57:53,426 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 16:57:53,428 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 17:03:05,746 :  Saving checkpoints for 72500 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 17:03:37,706 :  loss = 2.2375531, step = 72500 (344.290 sec)\n",
            "2019-05-09 17:03:37,709 :  global_step/sec: 7.26132\n",
            "2019-05-09 17:03:37,713 :  examples/sec: 929.449\n",
            "2019-05-09 17:03:37,717 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 17:03:37,718 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 17:08:50,076 :  Saving checkpoints for 75000 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 17:09:20,149 :  loss = 1.3545959, step = 75000 (342.443 sec)\n",
            "2019-05-09 17:09:20,154 :  global_step/sec: 7.30044\n",
            "2019-05-09 17:09:20,158 :  examples/sec: 934.457\n",
            "2019-05-09 17:09:20,161 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 17:09:20,164 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 17:14:32,580 :  Saving checkpoints for 77500 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 17:15:02,859 :  loss = 1.4262209, step = 77500 (342.710 sec)\n",
            "2019-05-09 17:15:02,862 :  global_step/sec: 7.29484\n",
            "2019-05-09 17:15:02,866 :  examples/sec: 933.74\n",
            "2019-05-09 17:15:02,871 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 17:15:02,872 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 17:20:15,222 :  Saving checkpoints for 80000 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 17:20:44,512 :  loss = 1.1135087, step = 80000 (341.653 sec)\n",
            "2019-05-09 17:20:44,515 :  global_step/sec: 7.31737\n",
            "2019-05-09 17:20:44,517 :  examples/sec: 936.623\n",
            "2019-05-09 17:20:44,525 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 17:20:44,526 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 17:25:56,919 :  Saving checkpoints for 82500 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 17:26:25,466 :  loss = 1.5507807, step = 82500 (340.954 sec)\n",
            "2019-05-09 17:26:25,469 :  global_step/sec: 7.33236\n",
            "2019-05-09 17:26:25,473 :  examples/sec: 938.542\n",
            "2019-05-09 17:26:25,478 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 17:26:25,479 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 17:31:37,834 :  Saving checkpoints for 85000 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 17:32:07,007 :  loss = 1.6306626, step = 85000 (341.541 sec)\n",
            "2019-05-09 17:32:07,010 :  global_step/sec: 7.31977\n",
            "2019-05-09 17:32:07,013 :  examples/sec: 936.93\n",
            "2019-05-09 17:32:07,016 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 17:32:07,017 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 17:37:19,450 :  Saving checkpoints for 87500 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 17:37:54,210 :  loss = 2.0486553, step = 87500 (347.203 sec)\n",
            "2019-05-09 17:37:54,213 :  global_step/sec: 7.2004\n",
            "2019-05-09 17:37:54,219 :  examples/sec: 921.651\n",
            "2019-05-09 17:37:54,223 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 17:37:54,224 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 17:43:06,613 :  Saving checkpoints for 90000 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 17:43:36,719 :  loss = 1.38201, step = 90000 (342.508 sec)\n",
            "2019-05-09 17:43:36,722 :  global_step/sec: 7.29909\n",
            "2019-05-09 17:43:36,728 :  examples/sec: 934.283\n",
            "2019-05-09 17:43:36,733 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 17:43:36,734 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 17:48:49,144 :  Saving checkpoints for 92500 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 17:49:20,153 :  loss = 1.1346642, step = 92500 (343.435 sec)\n",
            "2019-05-09 17:49:20,156 :  global_step/sec: 7.27941\n",
            "2019-05-09 17:49:20,157 :  examples/sec: 931.764\n",
            "2019-05-09 17:49:20,160 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 17:49:20,162 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 17:54:32,553 :  Saving checkpoints for 95000 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 17:55:05,720 :  loss = 1.2712753, step = 95000 (345.567 sec)\n",
            "2019-05-09 17:55:05,723 :  global_step/sec: 7.23449\n",
            "2019-05-09 17:55:05,727 :  examples/sec: 926.015\n",
            "2019-05-09 17:55:05,733 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 17:55:05,734 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 18:00:18,071 :  Saving checkpoints for 97500 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 18:00:45,702 :  loss = 1.2717034, step = 97500 (339.982 sec)\n",
            "2019-05-09 18:00:45,704 :  global_step/sec: 7.35334\n",
            "2019-05-09 18:00:45,712 :  examples/sec: 941.228\n",
            "2019-05-09 18:00:45,715 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 18:00:45,717 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 18:05:58,116 :  Saving checkpoints for 100000 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 18:06:27,653 :  loss = 0.8932342, step = 100000 (341.951 sec)\n",
            "2019-05-09 18:06:27,656 :  global_step/sec: 7.31097\n",
            "2019-05-09 18:06:27,663 :  examples/sec: 935.804\n",
            "2019-05-09 18:06:27,667 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 18:06:27,668 :  Dequeue next (2500) batch(es) of data from outfeed.\n",
            "2019-05-09 18:11:39,977 :  Saving checkpoints for 102500 into gs://bert_resourses/bert_model/model.ckpt.\n",
            "2019-05-09 18:12:07,905 :  loss = 0.53661036, step = 102500 (340.252 sec)\n",
            "2019-05-09 18:12:07,908 :  global_step/sec: 7.3475\n",
            "2019-05-09 18:12:07,910 :  examples/sec: 940.48\n",
            "2019-05-09 18:12:07,914 :  Enqueue next (2500) batch(es) of data to infeed.\n",
            "2019-05-09 18:12:07,915 :  Dequeue next (2500) batch(es) of data from outfeed.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N_OeXod-fHMT",
        "colab_type": "text"
      },
      "source": [
        "Training the model with the default parameters for 1 million steps will take ~53 hours. \n",
        "\n",
        "In case the kernel is restarted, you may always continue training from the latest checkpoint. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "77ZIAAATfzdF",
        "colab_type": "text"
      },
      "source": [
        "This concludes the guide to pre-training BERT from scratch on a cloud TPU. However, the really fun stuff is still  to come, so stay tuned.\n",
        "\n",
        "Keep learning!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RiFH_9Lbze5f",
        "colab_type": "code",
        "outputId": "5881a740-3e1d-4009-9724-0d0790e0cc39",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "!gsutil cp gs://hermes_assets/russian_uncased_L-12_H-768_A-12.zip gs://bert_resourses/"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Copying gs://hermes_assets/russian_uncased_L-12_H-768_A-12.zip [Content-Type=application/zip]...\n",
            "| [1 files][  1.9 GiB/  1.9 GiB]      0.0 B/s                                   \n",
            "Operation completed over 1 objects/1.9 GiB.                                      \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PP7_2pKWzfiZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}